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Abstract 


The Intelligence Reform and Terrorism Prevention Act of 2004 addressed the belief that 
weak analytic tradecraft had been an underlying cause of intelligence failures in the U.S. 
by requiring the Director of National Intelligence to establish and enforce tradecraft 
standards throughout the U.S. intelligence community (IC). However, analytic 
tradecraft—the innate abilities and learned skills of intelligence analysts, combined with 
the tools and technology needed to conduct analysis—is an understudied and poorly 
understood concept and a decade later, the frequency of intelligence failures has not 
improved. Using actor-network theory (ANT) as the foundation, the purpose of this 
qualitative narrative study was to gain greater clarity regarding the process of intelligence 
analysis and corresponding tradecraft. Data were collected through 7 semi-structured 
interviews from a purposely selected sample of U.S intelligence analysts to determine 
how they understood and navigated the analytic process. These data were inductively 
coded, and following the tenets of the ANT, the process and actors involved in 
transforming customer requirements and intelligence information into analytic products 
and refined collection requirements were identified and mapped. The central finding of 
this study is that current tradecraft standards address neither the full range of activities 
taking place nor the complete roster of actors involved in the analytic process. With this 
knowledge, the U.S. IC may be better positioned to identify specific training and 
equipment shortfalls, develop tailored reform efforts, and improve intelligence 
operations, resulting in potential positive social change. 



Analytic Tradecraft in the U.S. Intelligence Community 
by 

John J. Borek 

MS, National Intelligence University, 1996 
BS, The Pennsylvania State University, 1984 


Dissertation Submitted in Partial Fulfillment 
of the Requirements for the Degree of 
Doctor of Philosophy 
Public Policy and Administration 


Walden University 


July 2017 



Dedication 


I have been fortunate in my life to be surrounded by people who have loved me, 
supported me, and fostered my love of learning. Without a doubt, my parents and many 
of my teachers through the years have played an especially significant role in providing 
the foundation that helped me achieve this dream. Ultimately though, there is only one 
person to whom I could possibly dedicate the culmination of this effort, and that is my 
wife. Pamela has pushed, pulled, and carried me through this journey, and has had my 
back every step of the way. It is not an exaggeration to say that without her strength and 
spirit and belief in me, the words that follow in this dissertation would still be just 
random thoughts in my head and an unfulfilled desire in my heart. 



Acknowledgments 


I would like to take this opportunity to thank some of those who have helped me 
on this journey. First are my committee members, Dr. Bidjerano, Dr. Hirschbein, Dr. 
Campbell, and Dr. Gage. All have had their part in helping me become a more critical 
thinker, researcher, and writer. I especially appreciate Dr. B’s optimism and positive 
outlook and his certainty in this project and my ability to see it through. After every 
conversation or e-mail with him, I was ready to dig in and keep going again. No research 
effort based on narrative accounts would be possible without the willingness of 
participants to tell their story. To those who took time out of their day to respond to a 
request to participate in this study, I will always be grateful. I believe that your words 
will help others become more aware of the work of intelligence analysts as well as aid 
efforts to improve the profession of analysis. Working full time while pursuing any 
degree requires the cooperation of those paying one’s salary. Over the years, I have had 
no fewer than seven immediate supervisors, and all did whatever they could to help me 
balance the load. To my colleagues at work and my friends and family, all of whom have 
provided words of encouragement along the way, thank you. I hope you never 
underestimate the power of a positive word or a shared moment of enthusiasm. And of 
course to my wife, who has never doubted or let me forget that “with God all things are 
possible.” 



Table of Contents 


List of Figures.v 

Chapter 1: Introduction to the Study.1 

Introduction.1 

Background.2 

Problem Statement.3 

Purpose of the Study.5 

Research Questions.6 

Conceptual Framework.6 

Nature of the Study.9 

Definitions.10 

Assumptions.12 

Scope and Delimitations.13 

Limitations.13 

Significance of the Study.14 

Summary.16 

Chapter 2: Literature Review.17 

Introduction.17 

Literature Search Strategy.20 

Intelligence Analysis and the Actor-Network Theory - the Conceptual 

Framework.21 


Related Studies Using ANT. 


25 























Key Variables and Concepts.27 

Background of the U.S. Intelligence Community.27 

The Intelligence Cycle.29 

The Analytic Process.31 

Improving Analysis by Improving Inputs: Collection.38 

Improving Analysis by Improving Outputs: Analytic Transparency.39 

Improving Analysis by Legislation: The IRTPA.43 

Summary and Conclusions.49 

Chapter 3: Research Method.51 

Introduction.51 

Research Design and Rationale.52 

Methodology.54 

Participant Selection.54 

Instrumentation.56 

Data Collection and Analysis.57 

Procedures for Recruitment, Participation, and Data Collection.57 

Data Analysis Plan.58 

Role of the Researcher.59 

Trustworthiness.60 

Ethics and Confidentiality.63 

Summary.63 

Chapter 4: Results.64 

ii 

























Introduction. 


.64 


Data Collection.65 

Recruiting.65 

Interviews.66 

Data Analysis.67 

Implementing the Analytic Framework.67 

Coding and Analysis.69 

Results.70 

A Narrative Description of the Analytic Process.71 

Developing the Actor-Network of Intelligence Analysis.98 

Conclusions.108 

Evaluation of Trustworthiness.112 

Credibility.112 

Transferability.113 

Dependability and Confirmability.114 

Summary.115 

Chapter 5: Discussion, Conclusions, and Recommendations.117 

Introduction.117 

Interpretation of the Findings.118 

Limitations of the Study.121 

Recommendations for Future Research.122 

Implications.123 

iii 

























Intelligence Analysis and Tradecraft.123 

Actor-Network Theory and Tradecraft.125 

Conclusion.126 

References.128 

Appendix A: Interview Questions and Guide.149 

Appendix B: Participant Matrix.151 

Appendix C: Coding and Analysis Matrix.152 


iv 










List of Figures 


Figure 1. Graphic portrayal of the Intelligence Community’s Intelligence Cycle as 


described in The Church Committee Report .30 

Figure 2. Portrayal of the Intelligence Cycle in US National Intelligence: An Overview 

2013 .30 

Figure 3. The black boxing of the analytic process.31 

Figure 4. Graphic portrayal of Sherman Kent’s seven step analytic process as described in 

Strategic Intelligence for American World Policy .34 

Figure 5. DNI analytic standards and Kent’s analytic process.48 

Figure 6. Actor-Network Trace within the Analytic Black Box.98 

Figure 7. Detail Perspective of Actants in Ti.100 

Figure 8. Detailed Perspective of Actants in T 2 .103 

Figure 9. Detailed Perspective of Actants in T 3 .104 

Figure 10. Detailed Perspective of Actants in T 4 .106 


Figure 11. Detailed Perspective of Actants in T5. 


108 














1 


Chapter 1: Introduction to the Study 

Introduction 

The United States currently spends approximately $53 billion a year to support its 
national intelligence program (Office of the Director of National Intelligence [ODNI], 
201 la, 2012, 2013a, 2016). 1 Hundreds of thousands of people are employed in 
worldwide locations for the purpose of collecting and analyzing information “necessary 
for the conduct of foreign relations and the protection of the national security of the 
United States” (Dozier, 2013; ODNI 2011b, p. 7; Priest & Arkin, 2010). Research 
suggests the majority of Americans support a robust national intelligence program, and 
that national intelligence is “fundamental to America’s national security” (Obama, 2011, 
para. 23; Pew Research Center, 2015). Despite the criticality of the intelligence function, 
the amount of national capital invested in it annually, and the number of people putting 
forth their best effort on a daily basis, the national Intelligence Community (IC) still 
suffers from seemingly routine “intelligence failures.” 

Analytic tradecraft, which refers to the techniques used by intelligence analysts to 
do their jobs, has been cited in the literature, congressional hearings, and the press as a 
major reason for U.S. intelligence failures (Commission on the Intelligence Capabilities 
of the United States Regarding Weapons of Mass Destruction [WMD Commission], 
2005; Marrin & Clemente, 2006; Sanger, 2004). The perception of poor tradecraft in the 


1 This figure does not include money budgeted for those intelligence programs specifically identified to 
support military operations, known as the Military Intelligence Program. The combined national and 
military intelligence budgets peaked at $80.1 billion in FY 10 and was budgeted for $70.3 billion in in FY 
17. 
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IC generated enough interest that U.S. legislators specifically addressed it as an issue in 
the Intelligence Reform and Terrorism Prevention Act (IRTPA, 2004). However, based 
on my review of the literature, analytic tradecraft is shown to be an understudied and 
poorly understood concept. Because of this lack of knowledge, tradecraft reforms 
adopted as a result of legislative impetus have not changed the pattern of intelligence 
failures. Improving tradecraft may result in improved intelligence support to the United 
States and reduced intelligence failures. However, such improvement is unlikely in my 
opinion without greater understanding of the analytic process and associated tradecraft. 

In this chapter, I provide readers with background information necessary to 
understand my rationale for studying analytic tradecraft. Then, I state the problem I 
sought to address and present the research questions I designed to better understand 
tradecraft. After defining key terms, I introduce the theoretical framework I used and 
describe the nature of the study. I then consider the assumptions, scope and 
delimitations, and limitations of my research. I conclude the chapter by discussing the 
significance of the study and offering a summary of key points. 

Background 

The IRTPA, the most comprehensive reform of the IC since its creation, was a 
direct result of the 9/11 surprise attacks and the mischaracterization of the Iraqi Weapons 
of Mass Destruction (WMD) program (ODNI, n.d.). Developed under a broad bipartisan 
mandate, IRTPA reforms targeted the organizational structure and underlying 
responsibilities and authorities within the IC, technology supporting the enterprise, 
personnel management, and analytic tradecraft (Negroponte & Wittenstein, 2010; ODNI, 
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2006a). Reforming organizational structure involves adjusting government bureaucracies 
in order to achieve efficiencies and improve in management. Improvements to 
supporting technologies can range from the launch of a new satellite to the fielding of 
collaborative software for classified information networks. Personnel management, or 
human capital reforms, address the need to recruit, train, and retain quality personnel. 
These types of reforms have occurred in the IC since its creation and are generally well 
understood within government (Betts, 2002; Warner & McDonald, 2005). 

Reforming the analytic process by establishing community wide analytic 
tradecraft standards is a new approach and less well understood. Legislators and leaders 
of the IC components can identify and debate the strengths and weaknesses of different 
organizational relationships or the capabilities of a new technology. They struggle, 
however, to understand what actually constitutes the analytic process and analytic 
tradecraft (Marrin, 2011; Varouhakis, 2013). According to several authors, much of what 
is written and believed to be true about analytic tradecraft is based less on research and 
more on opinion or legend, representing a clear gap in the knowledge within intelligence 
studies (see Bruce & George, 2008; Johnston, 2005; Mangio & Wilkinson, 2008). I 
believe that a study specifically designed to explore and determine analytic tradecraft is 
needed before effective tradecraft reform can be implemented. 

Problem Statement 

Attempts to reform analytic tradecraft under the auspices of the IRTPA have not 
resulted in a reduction in the number of U.S. intelligence failures. Using Eiran’s (2005, 
p.4) understanding of an intelligence failure as having “a wrong picture of the 
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capabilities, intentions and threats of a hostile entity” and the gap between that perception 
and reality “exposed in an event that significantly compromises the national interests of 
the state;” in a review of academic literature, intelligence community professional 
journals, and press reports I identified 14 known cases of intelligence failure in the 57 
years between 1947-2004, or an average of approximately one failure every 4 years. In 
the 12 year period from the passage of the IRTPA in 2005 to the present (2017), there 
have been five significant and publicly identified intelligence failures. These are the 
surprise series of uprisings now known as the Arab Spring beginning in December 2010, 
the ability of Kim Jong-un to consolidate power in North Korea starting in 2011, the 
course of the Syrian war beginning in 2011, the rise of the Islamic State in Syria and Iraq 
beginning in 2013, and the Russian invasion of Crimea in 2014 (CBS, 2011; Koring, 
2010; Lake & Lily, 2014; Liptak, 2016; Muhammed, 2014; Robbins, 2015; Sanger. 

2014). Since the passage of the IRTPA the current trajectory of failures is actually on 
track to exceed the one failure every 4 years average. 

Addressing analytic tradecraft in the IRTPA represented a new approach to 
legislative oversight of intelligence. One assumption behind the IRTPA is that improving 
analytic tradecraft will result in fewer intelligence failures (Collins, 2004, p. 306). While 
this may be a valid premise, achieving those improvements without an understanding of 
what constitutes analytic tradecraft is unlikely. In the words of a senior ODNI official, 
post 9/11 analytic reform efforts have targeted “symptoms rather than root causes” 
(Neary, 2010, p.10). Within the IC and academia the understanding of the process of 
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analysis is still immature, compounding the uncertainty underlying analytic tradecraft. 
This has, in my judgment, limited the effectiveness of any potential reforms. 

Purpose of the Study 

The purpose of this study was to conduct research resulting in a better 
understanding of analytic tradecraft. Within the IC and academia, meaningful definitions 
of tradecraft or an understanding of the scope of analytic tradecraft seem to be lacking. 
For example, a 54 page government funded assessment of analytic tradecraft, which 
included 10 suggestions for improvement and a separate chapter of recommendations, did 
not include a definition or description of the term (Treverton & Gabbard, 2008). 

Gannon’s (1997) definition of “the special skills and methods required to do their 
business” (p. v) and Johnston’s (2005) more general “practiced skill in a trade or art” (pp. 
17-18) are vague, in my opinion. As if to address this issue, Johnston further stated that 
the term tradecraft has become a “catchall for the often idiosyncratic methods and 
techniques” (p. 17) of analysts, which purposely “obfuscates and complicates the reality 
of their [analysts] work” (p. 18). While not using the term tradecraft, Bruce and George 
(2008) identified seven essential skills an analyst must possess to be successful. These 
s ki lls combine the professional traits of “historian, journalist, research methodologist, 
collection manager, and professional skeptic” (Bruce & George, 2008, p. 3). At its core, 
intelligence analysis is the transformation of information into useful knowledge to reduce 
uncertainty and guide decision making (ODNI, 2013b, pp. 5-6). However, the focus of 
current analytic standards (Director of National Intelligence [DNI], 2007a; ODNI, 2015a; 
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Pigg, 2009) is on properly conveying analytic judgments, rather than the process used by 
analysts to arrive at their judgments. 

Research Questions 

In order to understand how analysts do their job, I posed the following research 
questions and subquestions: 

RQ1. How does an intelligence analyst move along the process of analysis, from 
becoming aware of the need for an intelligence product to creating the finished 
product? The subquestions for RQ1 were the following: How is the analytic 
process learned; specifically, does it involve formal training, trial and error, 
intuition; and does the process vary within and between individuals (and, if so, 
what explains the variance). 

RQ2. What software, hardware, tools, people, knowledge, and so forth populate 
the analyst’s environment, and how does the analyst negotiate the environment? 
The subquestions for RQ2 were the following: Does the environment affect the 
analytic process, and can the analyst manipulate the environment to match needs, 
or does the environment determine the process. 

RQ3. What skills (technical, social, or other) are needed to navigate the analytic 
environment? 

Conceptual Framework 

To arrive at a deeper understanding of analytic tradecraft, I used actor-network 
theory (ANT) as my conceptual framework. ANT was developed in the late 1980s within 
the sociology of science and technology field as a method of describing how knowledge 
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is created and shared in the scientific community (Latour, 2005; Law, 1992). ANT holds 
that knowledge “is the end product of a lot of hard work in which heterogeneous bits and 
pieces.. .are juxtaposed into a patterned network which overcomes their resistance” (Law, 
1992, p. 2). With its origin in medical research laboratories, the original actors identified 
were “test tubes, reagents, organisms.. .scanning electron microscopes, radiation 
monitors, other scientists.. .and all the rest” ), the developers also recognized that the 
approach might be useful in other aspects of the social sciences (Law, 1992, p. 2). 
Furthermore, ANT was a framework envisioned to be especially useful where there was 
little known about the environment being studied (Latour, 2005). Callon (1986) specified 
that ANT is appropriate as a means to study “a society which is considered to be 
uncertain and disputable” (p. 3). 

In describing the rationale for ANT, Law (1992) stated that “to understand the 
power of mechanics and organisation it is important not to start out assuming whatever 
we wish to explain” (p. 2). Agreeing, Latour (2005) noted “.. .instead of taking a 
reasonable position and imposing some order beforehand” (pp. 21 - 25) ANT provides 
the researcher a methodology that allows the actors to define their own environment. The 
researcher can then trace and describe the human, physical, ideological, and other 
connections that exist within the subject’s social network. Latour (2010) describes this 
environment, the social network, as a “complex ecology” in which an actor does not 
merely pass through as a self-contained entity, following the lines of a network diagram 
from one node to the next, but is instead sustained and transformed within its entirety. 



Using ANT requires the researcher to approach the study without an expectation 
of how the actor-network will develop. The three methodological principles of ANT are 
agnosticism, or removing any preconceived notions of the network; the use of a 
generalized symmetry in which every actor (human or non-human) in the network is 
considered an equal player; and the use of free association to establish the relationship 
between actors (Callon, 1986; Crawford, 2004; Delukie, 2009). When using ANT as a 
framework, “the search for order, rigor, and pattern is by no means abandoned” (Latour, 
2005, p. 23) but instead it is postponed until the actors themselves relate their role in the 
network, and the researcher can develop and trace the actions. This approach allows for 
an authentic portrayal of the process being studied from the perspective of the actor. 

Previous studies have used ANT as the conceptual framework to examine how 
actors operate within their environment to transfer knowledge and influence the decisions 
of others. Ranerup (2008) studied how public officials used an Internet-based decision 
support system to change how citizens considered their pensions and introduce pension 
reform. Czamiawska (2009) examined “institutional entrepreneurs” who transform 
existing organizations through the translation of ideas into energy and action. Weiss and 
Domingo (2010) used ANT to describe how technological innovations in newsrooms 
shaped how work was done and the resulting effect on products while Oh, Nam, and Kim 
(2016) used it as the framework in their study of knowledge based expert groups and the 
transfer of knowledge in the formulation of public policy. These efforts and others 
provide a strong foundation in applying ANT in the field of intelligence analysis. 
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For intelligence analysts, the environment in which they conduct analysis can 
include: the digital realm, such as several computer networks, each with unique software 
tools and security requirements; human interaction with customers, managers, editors, 
and other analysts; information, in various forms such as books, electronic message 
traffic, briefings, and collaborative web sites; and of course their own “wetware 2 ,” then- 
education, experiences, biases, and analytic techniques. Rather than beginning with a 
pre-ordained construct of analysis or tradecraft standards and fitting the experiences of 
analysts into it, ANT was used as the framework to capture the process as it occurs from 
the analysts’ perspective. The use of ANT to develop the actor - network of an analyst is 
a key component in developing a truer understanding of tradecraft, those skills and 
methods needed to successfully operate in the realm of analysts and perform the craft of 
analysis. 

Nature of the Study 

I designed this study using a qualitative narrative approach. A narrative is 
“understood as a spoken or written text giving an account of an event/action or series of 
events/actions, chronologically connected” (Czarniawska, 2004, p. 17). Narrative 
research, analyzing the detailed stories of individuals in their own words to understand 
specific experiences, is a commonly used within organizational studies (Creswell, 2007; 
Patton, 2002). If we want to better understand how analysts do their job, and the specific 


2 Defined in the Merriam-Webster as “the human brain or a human being considered especially with respect 
to human logical and computational capabilities.” 
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s ki lls and techniques they use to do it, having them inform us in their own words of the 
experience of creating an intelligence product is a sound approach. 

A narrative analysis can be guided by a conceptual framework (Creswell, 2007). 
The principles of ANT, combined with the rich description meaningfulness found in 
narratives, can combine to provide a description of the translation of information and 
knowledge into an intelligence product. Instead of beginning with an a priori description 
of tradecraft and asking analysts how they apply it or where they fit into it, by using the 
narrative approach the analysts were able to describe their environment and the activities 
therein in their own words. 

I used purposeful case sampling (Miles & Huberman, 1994; Patton, 2002) in order 
to provide a wide variation of experiences and more confidently address and describe the 
representative environment of analysts. My interviews were conducted with analysts 
currently working in IC organizations using an interview guide, in order to ensure each 
participant was queried on the same elements of their actor-network environment while 
maintaining the flexibility and freedom for them to use their own thoughts and words to 
define that environment (Patton, 2002). I transcribed the participant’s stories into text, 
then coded and organized this data following the tenets of ANT using MAXQDA 12, a 
data analysis software tool. 

Definitions 

A-Space : A collaborative work environment for IC analysts built on existing 
commercial architecture and fielded on a secure network. The Defense Intelligence 
Agency developed the software to foster discovery and communication between analysts 
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of different agencies in the IC (Jackson, 2009). In 2013 A-Space (analyst space) was 
replaced with an updated version known as i-Space (innovation space; (DoDIIS, 2012). 

Analysis: “The process, by which information is transformed into intelligence; a 
systematic examination of information to identify significant facts, make judgments, and 
draw conclusions” (ODNI, 2013b, p. 71). 

i-Space : A secure collaborative work environment in the IC (see A-Space). 

Intelligence: “Intelligence is information that agencies collect, analyze and 
distribute in response to government leaders’ questions and requirements” (Rosenbach & 
Peritz, 2009, p. 10). 

Intelligence Community: The group of 17 Executive Branch agencies and 
organizations that “work separately and together to engage in intelligence activities that 
are necessary for the conduct of foreign relations and for the protection of the national 
security of the United States” (ODNI, 2013b). The seventeen agencies and organizations 
are the Air Force Intelligence, Army Intelligence, the Central Intelligence Agency, Coast 
Guard Intelligence, the Defense Intelligence Agency, the Department of Energy, the 
Department of Homeland Security, the Department of State, the Department of the 
Treasury, the Drug Enforcement Administration, the Federal Bureau of Investigation, 
Marine Corps Intelligence, the National Geospatial-Intelligence Agency, the National 
Reconnaissance Office, the National Security Agency, Navy Intelligence, and the Office 
of the Director of National Intelligence (ODNI, 2013b). 

Intelligence cycle: The model used to explain the operation of the intelligence 
community. While different variations have existed since the mid-1970 ’s it is a five or 
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six step cycle that begins with the identification of a requirement by a customer and ends 
with the creation and dissemination of that product, leading to additional requirements. 
Analysis is a step on the cycle. 

Tradecraft : A term traditionally associated with espionage, it reflects both 
national and personal attributes of training, technology, and individual skill and cunning. 
During the era of analytic reform following the collapse of the Soviet Union, the concept 
of tradecraft was adapted to intelligence analysis to reflect associated skills and methods. 

Assumptions 

In conducting this study, I assumed that tradecraft was consistent among analysts 
and that it is not totally idiosyncratic or case specific. I also assumed that the analysts 
who volunteered to be study participants would be forthcoming. While tradecraft itself is 
not classified, the IC does exist in a world where secrecy and silence is the norm. For 
this reason, there was a possibility that analyst participants might be hesitant to talk in an 
unclassified research environment. 

I believed these assumptions were necessary in order to develop the interview 
guide and to use ANT to develop a more comprehensive understanding of tradecraft. 
Developing an accurate portrayal of tradecraft would not have been possible if tradecraft 
among analyst participants was individual rather than consistent. I would be unable to 
identify common transformation points and provide a trace of the analytic process in an 
analyst particular process. Further, if my participants deliberately provided false or 
misleading information, I also would have been unable to accurately assess and 
determine the phenomenon of tradecraft. 
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Scope and Delimitations 

The scope of this study and associated research questions was limited to 
individual tradecraft. Further, given the acknowledged dearth of an even basic 
understanding of analytic tradecraft, the intent was to focus on navigating the process of 
intelligence analysis holistically throughout the IC rather than trace the development of 
any one specific product or within one agency, precluding a more typical micro-level 
ANT approach (Cressman, 2009). Studies which use specific intelligence products or 
events, from Wohlstetter (1962) to the WMD Commission (2005), generally revolve 
around well-known failures. This bias toward examining failure, which while recurring 
and noteworthy, may mask more widespread tradecraft skills. 

Research previously conducted into the IC identified a community wide culture 
and also different agency cultures (Johnston, 2005; Nolan, 2013). While not discounting 
the effect that these sub-cultures may have on tradecraft, limiting the study to individual 
tradecraft is more likely to address the specific gap in knowledge identified and gain an 
understanding of the foundation of tradecraft. The sampling strategy maximized the 
cross section of the analytic population in order to mitigate any agency specific cultural 
effects. While narrative inquiries are not necessarily known for generalizability, the use 
of ANT as the framework and the sampling strategy allows for more confidence in 
transferring the results at a meso-level across the IC. 

Limitations 

As a member of the IC for over 30 years and a working analyst for at least half 
that time I bring my own personal biases on analytic tradecraft and the efficacy of 
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tradecraft reforms to improve analysis. While my experiences may allow me to better 
understand and share the culture and experiences of the research subjects, it may also 
limit my ability to interpret those experiences into generalizable findings. I chose ANT 
as my theoretical framework to mitigate those experiential biases; by adhering to the 
principles of ANT I conducted the research and analyzed the data without assuming I 
knew the actor-network of the analyst participants or that my approach to analysis was 
the correct or only one. Instead the framework enforced a methodology requiring the 
participants’ descriptions guide the results. 

While analysts operate in a classified environment, and the vast majority of their 
work requires access to classified information which results in duly classified products, 
the discussions on the process of analysis itself and accompanying tradecraft is 
unclassified. As an experienced and currently serving analyst in the IC, I developed the 
research questions to avoid the particulars of specific cases and instead elicited a 
narrative of the broader environment of the analyst and those skills needed to operate 
within it. By not focusing on a particular product or event with the analyst, instead 
drawing on more generalized observations and experiences, transferability of the findings 
were also enhanced. 

Significance of the Study 

Based on findings from the 9/11 Commission Report, the IRTPA mandated the 
Director of National Intelligence (DNI) address the issue of analytic tradecraft in the IC 
(IRTPA, 2004, 118 STAT6t. 3644, 3650). In response, the DNI has already instituted a 
number of IC-wide initiatives from training programs to new policies (ODNI, 2006a). 
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Addressing tradecraft issues without a clear understanding of the current state of the art 
risks creating the well intentioned “pathologies” that Betts (1978, pp. 84 - 85) warns can 
set the stage for the next failure, or at best develop irrelevant changes to procedures. A 
possible example of this type of hazard was the finding that, under the guise of enforcing 
tradecraft standards, a perception was created that analysis contradicting the United 
States Central Command narrative that Iraqi forces were successful in their fight against 
Islamic State forces was suppressed and modified (Inspector General, 2017). 

Community wide reform in the IC is costly, and while budgets have risen and 
fallen in the 13 years since the passage of the IRTPA the nation should not expect to fund 
reform efforts based on incomplete research and with limited expectations of success. 

Nor can the IC avoid the issue of analytic tradecraft in the quest to improve intelligence 
support. Many of the tradecraft standards first developed over 20 years ago following the 
collapse of the Soviet Union were mandated to the IC in community directives and still 
represent the extent of research into tradecraft today. As the IRTPA approaches 15 years 
of implementation there are no indications that these community tradecraft standards 
have improved analysis and reduced intelligence failures. A more comprehensive 
understanding of how analysts approach intelligence problems and operate within their 
environment will help to identify true shortfalls and tailor future policies and programs 
involving tradecraft reforms. The implications for positive social change with this 
understanding include an IC better positioned to improve intelligence support to the 
nation, and improved fiscal stewardship through the development of more focused 


reforms. 
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Summary 

In this chapter I provided the background needed to understand the rationale for 
this study of analytic tradecraft, the problem it addressed, and the research questions 
developed to improve our understanding of tradecraft. It described the nature of the 
study and introduced the conceptual framework guiding it. It also identified the 
assumptions used, limitations of the research and results, the significance of the study, 
and definitions of some key concepts. The following chapter will provide a review of the 
current literature and scholarship on intelligence analysis and analytic tradecraft. 
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Chapter 2: Literature Review 

Introduction 

In this review I identify, analyze, and synthesize the literature on analysis in the 
U.S. intelligence community. Both the 9/11 and Iraqi WMD commissions identified 
weak analysis and analytic tradecraft as a contributing cause of those intelligence failures 
(9/11 Commission, 2004, pp. 339, 344 - 348; WMD Commission, 2005, p. 12), and 
legislation ensuing from those studies specifically mandated improvements in tradecraft 
(IRTPA, 2004, 118 STAT6t. 3644, 3650). By reviewing what is known and believed to 
be true about intelligence analysis and its corresponding tradecraft, and by identifying 
gaps in that knowledge, we can begin to develop a more comprehensive understanding of 
the analytic process. 

The literature of intelligence analysis falls into three broad categories: 
professional journals, books, and training materials within the intelligence community, 
which had limited distribution and were in many cases classified until the 1980s; 
government publications and testimony generally associated with the results of audits, 
investigations, or other external inquiries into the working of the intelligence community; 
and professional journals and books in the social sciences and political sciences which 
intersect with the work of the intelligence community (e.g., work in international 
relations). These three types of literature are included in this review, as well as formerly 
classified material, where available. 

There is a wide range of meaning to the concept of analysis itself. The Merriam- 
Webster dictionary provides seven discrete definitions for analysis, many of which have 
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more than one element within the definition (Analysis, 2013). To me, none seem s 
particularly suitable in a national security context. The DNI definition of analysis (“the 
process by which information is transformed into intelligence; a systematic examination 
of information to identify significant facts, make judgments, and draw conclusions” 
[ODNI, 2013b, p. 71]) provides more specificity to the expected result but still allows 
considerable latitude in understanding how the process occurs. 

The architect of analysis in the U.S. IC, Sherman Kent, begins “one of the most 
influential books ever published on intelligence analysis” (Wagner, 2002, p.l) by stating 
that “[ijntelligence is a simple and self-evident thing.. .intelligence work is in essence the 
search for the single best answer” (Kent, 1949, p. vii). He went on, however, to identify 
“the three separate and distinct things that devotees of intelligence usually mean when 
they use the word” (Kent, 1949, p. ix) in the context of national security: knowledge, 
organization, and activities. Organizing his book along those three elements of national 
intelligence, Kent wove discussions on analysis through each section. 

Kent presciently identified the controversy that would resist settlement over 60 
years later: while the role of the analyst to provide intelligence to reduce uncertainty is 
obvious, how the analyst arrives at that information is ill defined and straddles several 
conceptual boundaries of intelligence. Analysis is understood to be “the thinking part of 
the intelligence process” (Bruce & George, 2008, p. 1). The process alluded to by the 
ODNI definition of analysis (ODNI, 2013b, p. 71) is the transformation of disparate bits 
of information into useful knowledge at the request of a customer in order to reduce 



19 

uncertainty and guide decision making. The analysts’ role and that of corresponding 
tradecraft occupy a complex position in the larger IC. 

As I will demonstrate in this literature review, the vagueness behind 
understanding the process of analysis and how analysts navigate that process directly 
affects the understanding of analytic tradecraft. Using a concept found in science and 
technology studies, and ANT, it is clear that the process of analysis was black boxed. A 
black boxed process is one in which the inputs and outputs of the process could be 
identified and even measured, but there was no need or desire to understand the process 
itself because it was either too complex, mysterious, or obscured (Bell, 2006; Stowell & 
Welch, 2012). The lack of detailed understanding of the analytic process by legislators 
and IC components has shaped efforts to improve analysis. 

The back-to-back national tragedies and intelligence failures of the 9/11 attacks 
and the mischaracterization of the Iraqi WMD program and subsequent invasion of Iraq 
resulted in an unprecedented overhaul of U.S. intelligence. This set of reforms included, 
for the first time in a long history of intelligence reforms, an attempt to mandate by 
legislation the inner workings of the black box by establishing analytic tradecraft 
standards and mandating the DNI enforce and evaluate compliance with those standards 
(Rosenbach & Peritz, 2009; Warner & McDonald, 2005). Without a clear understanding 
of what analysts do however, it is my opinion trying to improve analysis and analytic 
tradecraft will necessarily be a haphazard endeavor with uncertain results, evidenced by 
failure to improve the rate of intelligence failure. 
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In addition to reviewing the literature on intelligence analysis, this chapter 
includes a review of the literature relevant to the conceptual framework and methods I 
used in this research effort and its applicability to a study of tradecraft. While ANT has 
not been used as a framework in previous studies in the IC, other researchers have used it 
to develop a deeper understanding of knowledge creation and spread in other fields. The 
following sections include a review of the conceptual framework used in the research, a 
brief background on the creation of the U.S. IC, and then a review the literature that 
examines analysis, the role of analysts, and previous attempts to address issues of 
tradecraft. 

Literature Search Strategy 

I obtained the materials supporting this literature review from a wide variety of 
physical and online resources. Walden University Library’s journal, e-book, and thesis 
and dissertation databases provided the initial results when I used broad search terms 
such as intelligence analysis, analysis and tradecraft, and intelligence community, 
analysis, and reform. Searches on Google Scholar identified literature missing from 
Walden Library databases. As I identified authors and subject matter experts in both 
original material and as references I conducted name searches for similar antecedent or 
follow-up research efforts. 

The Government Accountability Office and other government oversight 
committee websites were used a resource for previous government studies of intelligence 
reform and analytic tradecraft. As a working analyst, I was able to use my own agency’s 
library as a source for materials on analysis and tradecraft. In addition, the National 
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Defense University library was used as a resource for limited production documents 
identified as references in internal IC materials. I used the Central Intelligence Agency’s 
Center for the Study of Intelligence as a resource for previously classified materials and 
articles published in IC specific journals. My physical proximity to the University of 
Virginia also allowed me to use that facility to search for and obtain materials. 

An important restriction on my intelligence and tradecraft related searches to note 
is that classified materials leaked to the public and available through various online 
resources were not searched or used in this research effort. As a cleared and active 
analyst, I know that using classified material in a manner which does not protect it from 
unauthorized disclosure, regardless of its availability or how I obtain it, is a criminal 
offense. For this reason, only officially declassified or redacted and released government 
materials were used. 

In addition to reviewing the literature on intelligence analysis and tradecraft, I 
also searched for materials on ANT. I conducted searches involving both the theoretical 
background and development of that framework, and applications of ANT in previous 
research efforts. The majority of ANT related materials were obtained through Walden 
University’s library, the University of Virginia’s library, ANT specific websites, and 
Google Scholar. 

Intelligence Analysis and the Actor-Network Theory - the Conceptual Framework 

The conceptual framework I used to guide this study was ANT. ANT was 
originally developed as a means to understand how the scientific community creates 
knowledge; “knowledge” defined to be the end result of a social effort in which a natural 
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tendency towards isolation is overcome to combine disparate and heterogeneous elements 
into some material form (e.g., papers, presentations, transferrable skills) (Law, 1992). 

The ANT approach is a means of allowing the participants in a social effort to describe 
their environment and their activities in their own words, and then trace, or map, the 
assembled network from their own descriptions (Latour, 2005). The parallel between the 
creation of knowledge in the scientific community and the creation of knowledge in the 
intelligence community provides an appropriate case for the application of ANT in this 
study. 

ANT is not a theory in a classic sense, it does not offer an explanation of why a 
particular network of action forms the way it does, or allow for a prediction of a 
network’s structure; it is instead a method for describing and understanding actions 
within networks (Fenwick & Edwards, 2011; Latour, 2005). ANT guides the researcher 
in exploring new, previously unstudied, or understudied social activities and then 
developing a cogent description of the resultant network. The resulting network from an 
ANT guided study is not the trace of a physical infrastructure of nodes and paths, but a 
means to put a form to the “phantoms” of social activities by unpacking previously 
assumed self-contained events into actions, actors, and transformations (Latour, 2010) 

Bruno Latour, one of the co-founders of ANT, in antecedent research conducted 
an anthropological study of a Salk Institute laboratory over a two year period examining 
the construction of scientific facts (Latour & Woolgar, 1979). Latour observed that 
scientific research, “a body of practices widely regarded by outsiders as well organized, 
logical and coherent in fact consists of a disordered array of observations with which 
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scientists struggle to produce order” (p. 36). Latour’s work in the laboratory shaped his 
thoughts on the role of the social environment in the scientific community. Fellow 
researchers and staff, the laboratory equipment used, reputations of individual researchers 
and facilities, and many other social interactions were all factors in how the scientists 
arrived at their conclusions and how the results were presented. None of this information 
is ever captured by the articles published on the research effort or ensuing results, the 
hallmark of a successful research project; in fact identifying relevant social factors in 
scientific endeavors can be used to cast doubt on the findings resulting in an incomplete 
understanding of the true scientific process (Latour & Woolgar, 1979). 

Latour, in partnership with John Law, Michael Callon, and others built on this 
background through the 1980’s resulting in ANT (Cutcliffe, 2000; Law, 1992). There are 
three methodological principles of ANT: agnosticism, or removing any preconceived 
notions of the network; the use of a generalized symmetry in which every actor (human 
or non-human) in the network is considered an equal player; and the use of free 
association to establish the relationship between actors (Callon, 1986; Crawford, 2004; 
Delukie, 2009). A clearer understanding of the process and supporting infrastructure can 
be developed by researchers using this approach. When using ANT as a framework “the 
search for order, rigor, and pattern is by no means abandoned,” but instead it is postponed 
until the actors themselves relate their role in the network and the patterns they describe 
can be traced (Latour, 2005, p. 23). 

ANT holds that knowledge “is the end product of a lot of hard work in which 
heterogeneous bits and pieces.. .are juxtaposed into a patterned network which overcomes 



24 


their resistance” (Law, 1992, p. 2). While the diverse elements originally driving the 
development of ANT were “test tubes, reagents, organisms... scanning electron 
microscopes, radiation monitors, other scientists.. .and all the rest” (Law, 1992, p. 2), the 
developers also recognized that the utility of the approach in other aspects of the social 
sciences. Further, ANT was a framework specifically envisioned to be used where there 
is little known about the environment being studied (Latour, 2005). Callon (1986) 
specifies that ANT is appropriate as a means to study “a society which is considered to be 
uncertain and disputable” (p. 3) while in a defense of ANT Latour (1999) reminds us that 
“actors know what they do and we have to learn from them not only what they do, but 
how and why they do it” (p. 18). 

In describing the rationale for ANT, Law (1992) states that “to understand the 
power of mechanics and organisation it is important not to start out assuming whatever 
we wish to explain” (p. 2). Agreeing, Latour (2005) notes “.. .instead of taking a 
reasonable position and imposing some order beforehand” (pp. 21 - 25) what ANT 
provides is the researcher a methodology for allowing the actors themselves to define 
their environment. The researcher can then trace and describe the connections, human, 
physical, ideological, etc.; that exist within the subject’s social network. He describes 
this environment as a “complex ecology” in which an actor doesn’t merely pass through 
as a self-contained entity, following the lines of a network diagram from one node to the 
next, but is instead sustained and transformed within the entirety of the pathway (Latour, 


2010). 



25 


I will present the position in the following sections that the parameters of 
tradecraft in place today, as defined from the legislation and implementing instructions, 
have for the most part been decided a priori. By following the concept of analytic 
tradecraft to its origin we see that the intelligence community’s current understanding of 
tradecraft is the result of an effort to address the perception of a specific problem - the 
politicization of intelligence regarding the Soviet Union during the Cold War. Asa 
community of practitioners, the IC’s true understanding of the analytic process and 
resultant analytic tradecraft is still immature however. This thought is captured by 
Marrin (2011) when he states that “where there has been a lot of general descriptions of 
the analytic process, there is very little detailed scholarship that describes exactly who 
analysts are and exactly what they do on a daily basis” (p. 9). This black boxing of the 
analytic process, where it has essentially been neither studied nor described, (Broer, 
Nieboer, & Bal, 2010) is a situation similar to others which benefited from ANT guided 
studies as described below. 

Related Studies Using ANT 

My review of the literature has not identified the use of ANT in the study of 
intelligence analysis or the intelligence community. ANT has however been used as the 
guiding framework to examine the complex ecology of individuals involved in 
developing patterns of behavior within organizations and transforming information into 
knowledge. For example, ANT has been used extensively in the education field 
(Fenwick & Edwards, 2012) and in sociological studies of science and technology 


(Bijker, Hughes, & Pinch, 2012). 
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Czarniawska (2009) used ANT to guide a study on the emergence of institutions 
in organizations. Focusing on the transformations that take place with an organization, 
Czarniawska examined how entrepreneurs within an organization conduct activities 
which can then become routinized and further translated into durable institutions. Using 
narratives Czarniawska was able to trace and develop an understanding of the spread of 
activities within the London School of Economics. 

Ranerup (2008) used ANT to examine the role of technology, specifically an 
Internet- based decision support system, in shaping opinions and follow on actions. 
Using the success of the revamped Swedish public pension system as the focus, Ranerup 
interviewed seven individuals associated with the development and fielding of the 
software to understand their views and understanding of the project. With the goal of 
simplifying complex information for non-experts (i.e., working individuals managing 
their pension funds), ANT was used to determine how an actor’s understanding of the 
needs and capabilities of their consumer shaped their activities. 

Psenaka (2008) used ANT as the theoretical framework to bring understanding to 
the complex heterogeneous network that is NASA’s manned spaceflight maintenance 
program. Building on Latour’s earlier work, Psenaka demonstrated that knowledge was 
not something “originating in the minds of experts that can be coded and then shared 
through discourse” but “an accomplishment and outcome of interaction within 
associations of people and things” (2008, p. 43). Describing the maintenance process as 
comprised of knowledge objects (e.g., people, paper, parts, and places) and knowledge 
practices, ANT was used to describe the “knowledge space” where these components 
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came together in a stable and repeatable environment to successfully maintain spacecraft 
(Psenaka, 2008, p. 28). 

Oh et al. (2016) used ANT as the theoretical framework in their study of 
knowledge creation during the process of policy making in the Seoul, South Korea. As 
the city leaders of Seoul began an effort to develop “women friendly” policies addressing 
five different categories of city living, they assembled a panel of experts to advise and 
shape the development of the specific projects. In this case, ANT was used to identify 
how individual tacit knowledge was transformed into group explicit knowledge resulting 
in specific civic projects. 

Weiss and Domingo (2010) used ANT to trace the complex ecology of 
newsrooms, and to examine how the introduction of new technology changed 
relationships and activities within the network. The authors describe how the black box 
of journalism was previously opened and explored, but that fundamental changes in those 
networks occurred as a result of new technology and business practices. This study, 
which used four cases of on-line newsrooms, highlighted the importance of the 
relationship between journalists and their supporting information technology staff. 

Key Variables and Concepts 
Background of the U.S. Intelligence Community 

A standing intelligence apparatus is a relatively recent innovation in US national 
security. Prior to WW II national intelligence was a segmented activity in government 
and a primarily overt function, especially in peacetime, infamously articulated by 
hawkish Secretary of State and War Stimson in his comment that “gentlemen do not read 
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other gentlemen’s mail” (National Security Agency [NSA], 2009). U.S. State 
Department foreign officers would observe and report on the politics and economics of 
the country in which they served, while the military departments maintained attaches 
which would observe and report on developments occurring with foreign militaries. 

There was little to no coordination between the agencies, even between the Army and 
Navy. Roberta Wohlstetter’s (1962) classic intelligence study Pearl Harbor: Warning 
and Decision describes the environment found in pre-WW II U.S. intelligence elements, 
to include scant resourcing, parochial production and limited dissemination, inter-service 
rivalries, and the lack of training or professional development. During periods of crisis 
ad hoc committees were formed combining diplomats, military experts, federal law 
enforcement personnel, and outside experts. When the precipitating crisis was over the 
committees would disband and the peacetime status quo would return (Marrin, 2006; 
NSA, 2009). 

The concept of tradecraft within the sphere of national intelligence has 
traditionally involved the nuts and bolts of espionage, or spying. Whether professional, 
academic, or fiction the literature of tradecraft might typically include practical 
techniques for safely meeting with agents (Konovalov & Sokolov, 1960), developing and 
using dead drops (Haberstich, 1966), lock picking (Wise, 1986), or in more current 
literature, a review of how the 9/11 terrorists avoided detection prior to their attacks 
(National Commission on Terrorist Attacks Against the US [9/11 Commission Report], 
2004). An intelligence officer’s tradecraft was a reflection of both national and personal 
attributes, of training, technology, and individual skill and cunning. 
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In the period of reform following the fall of the Berlin Wall and collapse of the 
Soviet Union, the idea of tradecraft was purposely migrated to the analytic side of the IC 
to “capture our special skills and methods” (Gannon, 1997, p. v). Unfortunately, those 
skill and methods remained largely undefined despite attempts to identify and propagate 
best practices. It was relatively easy to identify a specific component of analysis as 
tradecraft, structured analytic techniques for example, but an understanding or definition 
of tradecraft which encompassed the whole of the analytic process and described how the 
elements fit together was never developed. 

The Intelligence Cycle 

The intelligence cycle has been used to describe the activities of the IC since first 
succinctly articulated in the 1976 Church Committee Report; interestingly however, the 
report also found the realities of community operations “barely recognizable” when 
compared to the model (Select Committee to Study Governmental Operations [Church 
Committee], 1976, Ch. 2, pp. 17 - 19). Arguably, the steps had first been described 
generally by Kent in 1949 as the research process and are covered in greater detail later in 
this chapter. Adapted to encompass the operations of the entire IC and the roles and 
missions that emerged as the community grew and matured, the intelligence cycle had 
lost and then regained process steps over the years as different components are 
emphasized but has remained essentially unchanged since 1976. 
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Figure 1. Graphic portrayal of the intelligence community’s intelligence cycle as 
described in The Church Committee Report (1976, pp. 17 - 19). 



Figure 2. Portrayal of the intelligence cycle in US National Intelligence: An Overview 
2013 (2013, p. 4). 


Regardless of which version of the intelligence cycle referenced since 1976, 
analysis has always been identified as a step in the process. As I stated previously, and 
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will explore in more detail below, how that analysis is conducted, what it entails, and the 
environment in which it is performed, has essentially been black boxed (Bell, 2006; 
Michael, 2000; Stowell & Welch, 2012). The input and output of the analytic process are 
clearly understood and can even be measured and evaluated. Information, colloquially 
known as reporting, and customer requirements are fed into analytic process; w hi le 
finished intelligence products and requirements for additional intelligence collection to 
fill identified gaps are the resultant output. Yet, the analytic process itself remains 
hidden, and little, if any, attention is given to it as long as the input and output continues 
as expected (Borek, 2014). 


- Requirements 

- Reporting 


ANALYTIC PROCESS 


- Products 

- Information Gaps 


Figure 3. The black boxing of the analytic process. 


The Analytic Process 

Following WW II, the fall of the Iron Curtain, and the existential threat posed by 
nuclear weapons and intercontinental delivery systems, the nation’s leadership decided to 
create a permanent intelligence organization which could act as a standing corporate body 
providing warning of attack and would be immediately prepared to respond to any crisis 
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situation. In addition to revamping the structure of the Armed Forces, the National 
Security Act of 1947 built upon President Truman’s January 1946 memo ordering the 
coordination of existing foreign intelligence activities (National Security Act, 1947; 
Truman, 1946) - more specifically the analysis and production of intelligence. In the 
1946 - 1947 time period these two documents demonstrate that the President and his 
advisors identified the need to coordinate and professionalize analysis was a fundamental 
consideration as the U.S. intelligence community was designed. 

Sherman Kent, a history professor at Yale University, was one of those outside 
experts brought into national service during times of crisis. In WW II he served in the 
Research and Analysis Branch of the Office of Strategic Services (OSS) (Steury, 1994). 
Following the end of the war he returned to Yale in 1947 where he wrote Strategic 
Intelligence and American World Policy. In the preface to Strategic Intelligence he 
described that while intelligence was now more critical than ever to preserve national 
security it also suffered from “confusions” among and between the producers, users, and 
beneficiaries [i.e., citizens] of intelligence; confusions rooted in “imprecisions which 
have grown up in the language of intelligence and which have found permanence in the 
manuals” (Kent, 1949, p. ix). In Strategic Intelligence he defined for the members of the 
nascent national intelligence organization what intelligence was and how to create it. He 
described national intelligence as comprising three elements: knowledge, organization, 
and activity (Kent, 1949, p. ix). 

Kent (1949) further described intelligence as a combination of surveillance and 
research, not individual pieces of information but knowledge gained through a process, 
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identifying that “surveillance without its accompanying research will produce spotty and 
superficial information” (pp. 151-155). This analytic process, which he called research, 
was the only way that analysts could provide . the truth, or a closer approximation to 
the truth, than we now enjoy” (Kent, 1949, p. 155). 

While Kent (1949) drew distinctions between different types of intelligence and 
intelligence products, he identified that all intelligence was derived from the same 
process. With his background in Ivy League academia, Kent acknowledged that the 
research process he describes for analysts approximates academic research in the social 
sciences, yet he also distinguished it from academic research because of the types of 
issues driving intelligence work and the nature of the data available to analysts. He 
ascribed the following seven steps to the analytic process: 

1. The appearance of a problem requiring the attention of the intelligence 
staff. 

2. Analysis of the problem to discover which facets of it are of actual 
importance and which of several lines of approach are most likely to be 
useful. 

3. Collection of data bearing upon the problem defined in the step above. 
This step involves a survey of data already available and those efforts to 
procure new data to fill in gaps. 

4. Critical evaluation of the data assembled. 

5. Study of the evaluated data with the intent of finding some sort of inherent, 
meaning. Kent called the discovery of such meaning “the moment of 
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hypothesis.” He also acknowledged that “there is also rarely such as thing 
as one moment of hypothesis” and that it can occur anytime, if not at 
multiple times during the process. 

6. Continued collection of data focused by the more promising hypotheses in 
order to confirm or deny them. 

7. Establishment of one or more hypothesis as truer than others and 
statement of these hypotheses as the best approximations of truth, what he 
identified as the presentation stage. (Kent, 1949, pp. 157 - 158) 


The Analytic Process - Kent 



Kent, S. (1949). 

Figure 4. Graphic portrayal of Sherman Kent’s seven step analytic process as described 
in Strategic Intelligence for American World Policy (1949, pp. 157 - 158). 

Kent described what he saw as the analytic process - from the perspective of an 
analyst - at the dawn of the IC. After reading Strategic Intelligence and leading the IC 
through the intelligence failure associated with the outbreak of the Korean War, the 
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director of the recently created Central Intelligence Agency requested Kent return to 
government service in 1950 to develop a newly envisioned Board of Estimates. Kent 
agreed, and remained with the CIA until his retirement in 1967 (Davis, 2002). 

An understanding among scholars and practitioners that the analyst held a 
position in the analytic process stretching from the initiation of the information request 
through presentation can be found from Kent’s initial description of research through 
today. Drell (1957) described basic approaches to the research process, explicitly 
identifying similarities and differences between intelligence analysis and academic 
research and developing four steps for the process: project planning, data collection, data 
analysis, and presentation. Drell also includes in this analytic primer advice on how an 
analyst must take into account the emergent office automation available at the time as 
part of the research process. Expanding on the idea that analysis is a process, and that 
analysts are required to be aware of and involved all aspects of the process, Borel (1959) 
offers a pre-digital age primer for analysts on the strength and limitations of various data 
storage and retrieval systems. In a case study of the first attempts to develop a repeatable 
methodology to forecast the German Army’s ability to supply itself on the Eastern Front 
in 1941, Quibble (1962) describes how the scholar-analysts of the day were involved in 
every step of the analytic process, from framing the question and assumptions, 
identifying the variables examined, and developing sources and collecting archival 
information. 

A 1990 US Army Field Manual on Intelligence Analysis describes analysis as a 
“system” (Headquarters, Department of the Army [HQDA], 1990, p. 1-6) and discusses 
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the role of the analyst in each step of the intelligence cycle. It states that the processing 
phase of the intelligence cycle, “in which information becomes intelligence,” contains 
three steps: recording, evaluation, and analysis (HQDA, 1990, pp. 2-3 - 2-4). An 1994 
Army training manual for intelligence analysts identifies the analytic occupational 
specialty as a “jack of all trades” and a sampling of the expected duties an analyst is 
expected to perform includes “.. .knowledge and production of enemy 
doctrine.. .collection of intelligence information, processing and recording information, 
producing intelligence reports, and dissemination of those reports” (U.S. Army 
Intelligence Center and School [U.S. Army], 1994, p. 1-2). 

The Dulles Report, a review of national intelligence undertaken soon after the 
implementation of the National Security Act of 1947, was critical of the lack of 
coordination of analytic products between the newly created CIA and the other agencies 
of the IC, and between the analysts of the different agencies themselves (Warner & 
McDonald, 2005). While not explicitly stated, the report implied that analysts should not 
be relegated to isolation but instead active participants within their own agencies and 
across the IC. Completed in 1949, recommendations of this report were adopted in 1950 
following President Truman’s removal of the serving Director of Central Intelligence as a 
result of the two critical failures of the Korean War - first to incorrectly forecast the 
intentions of North Korea to invade and second the failure to identify the intent and 
execution of the infiltration of 200,000 Chinese troops sent to aid North Korea. 

In 1992 Robert Gates, then serving as Director of Central Intelligence, responded 
to charges of politicization in agency’s analysis as a factor in failing to accurately assess 
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the health of the Soviet Union, providing insight into his perspective of analysis. He 
described intelligence as a process, and discussed the desired relationship between 
analysts and policy makers throughout that process. Gates went on to specify that 
analysts must be trained to “.. .gather evidence, assess sources, make judgments, and 
write up or brief their analysis, our so-called tradecraft” (Gates, 1992a). At this early 
stage of identifying a tradecraft associated with analysis, Gates saw it as spanning the 
length of Kent’s research model of analysis. 

Moving forward in time, contemporary scholars and practitioners provide little 
new insight into the analytic process. Wheaton and Beerbower (2006) define intelligence 
as “a process focused externally using all sources that is designed to reduce the level of 
uncertainty for a decision maker” (p. 329) and build a strong case for an analytic process 
embedded throughout the larger intelligence process; although they point out that 
additional research needs to be done on the “open question of process” (p. 329). Bruce 
and George (2008) also take a wide view of the analytic profession, identifying seven 
essential skills a successful analyst must possess to be successful: mastery of the subject 
as well as U.S. policy relative to their specialty; an understanding of research methods 
and ability to apply them; a combination of imagination and scientific rigor in order to 
develop and test hypotheses; an understanding of the collection system; an awareness of 
personal cognitive biases and influences; a willingness to consider alternative hypotheses 
of the data; and enough self-confidence to admit and learn from mistakes. Bruce and 
George (2015) later go on to argue for taking additional steps to professionalize the 
career field of intelligence analysis, in which they define tradecraft as “the methodology 
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of intelligence analysis,” (pp. 1-2) while acknowledging that the IC suffers from a 
culture which considers training and skill development to be a “secondary priority” (p. 
22 ). 

Improving Analysis by Improving Inputs: Collection 

The lessons of the Korean War failures and the ensuing entrenchment of the Cold 
War (NATO military headquarters was created in 1951, the Warsaw Pact in 1955) began 
the emphasis on developing technology to penetrate denied areas. The 1955 Hoover 
Commission Report on Government Organization, Intelligence Activities Report, 
advocated for stronger collection efforts in the IC, stating that “collection of intelligence 
is a vital element in the fight to preserve our national welfare and existence” and found 
instances where “too conservative an attitude” had resulted in missed opportunities 
(Commission on Organization of the Executive Branch of Government [Hoover 
Commission], 1955, p. 43). Increasing the flow of information available to analysts by 
technical means was seen as a means to improve the analytic product. 

Ironically, in 1966, just eleven years after the publication of the Hoover 
Commission report, a CIA Inspector General (IG) report found that the IC was 
“collecting too much information.. .flooding the system with secondary 
material.. .degrading production” (Central Intelligence Agency, 1974, p. D-2). Analysis 
was suffering not only because analysts had more information to evaluate, but the flood 
of data was changing how analysts approached their job. The report found that the 
emphasis on collection had resulted in a “jigsaw theory” of intelligence; a belief that by 
collecting one more missing piece of information a puzzle could be solved (Church 
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Committee, 1976, Ch. 12, pp. 274 - 275). In 1970 President Nixon, disappointed with 
the quality of intelligence he was receiving, authorized a study of the IC which resulted in 
“A Review of the Intelligence Community” published in 1971, better known as the 
“Schlessinger Report.” This report identified that the quality of analysis had not risen 
despite access to large amounts of data, and in fact, the IC had not “shown much 
initiative in developing the full range of possible explanations in light of available data” 
(Schlessinger, 1971, p. 10). The report also concluded that “there is a strong presumption 
in today’s intelligence set-up that additional data collection rather than improved analysis 
will provide the answer to particular intelligence problems” (Schlessinger, 1971, p. 11). 
Because of the focus on increasing collection and improving the technology behind 
collection assets, analysis and production had become “the stepchild of the community” 
(Schlessinger, 1971, p. 11). The 1966 CIA IG report and the 1971 Schlessinger Report 
identified that the analytic profession did not have strong career incentives within any 
element of the IC, that analysts were faced with confusing or conflicting priorities, that 
the relationship between analysis and collection was backwards, and that analytic 
practices in general were poor and the analysis itself was shallow (Central Intelligence 
Agency, 1974; Schlessinger, 1971). 

Improving Analysis by Improving Outputs: Analytic Transparency 

As the IC grew, and the amount of information available to analysts expanded, the 
advanced technology that enabled that collection created distance between analysts and 
collectors (Goodman, 1984). Analysts, whom Kent envisioned in all three domains of 
national intelligence and also throughout the research process, appeared to become 
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sequestered within the community into one step of the intelligence cycle. Summing up 
the perception of the era, Peter Szanton and Graham Allison (1976) characterized 
inadequate analysis as the number one deficiency in the IC and the analytic process as 
“data collected by sophisticated methods and analyzed by primitive ones” (p. 184, 190). 
They argued that flaws in the analysis included “bias, irrelevance, and a judgmental 
rather than analytic orientation” (Szanton & Allison, 1976, p. 190). They advocated 
increasing the number and quality of analysts in the IC and encouraging an atmosphere of 
competitive analysis between agencies to improve analysis overall. 

Simultaneously, a seemingly growing list of failures in analysis, culminating in 
the failure to warn of the fall of the Berlin Wall and the subsequent collapse of the Soviet 
Union, brought on the impetus to open the black box of analysis and tweak the contents. 
Charges of politicization of analysis was given as a cause for those analytic failures 
during the nomination hearings of Robert Gates for Director of the CIA (Gates, 1992b). 
During the hearings the need to address analytic procedures and make the analytic 
process more transparent in the future became an accepted fact and resulted in a focus on 
improving analytic techniques and providing clearer output to the consumer. It was 
during these hearings that Douglas MacEachin, an analyst whose views on the Soviet 
Union differed from those of Gates, testified to Gates’ analytic integrity and willingness 
to consider and publish alternative analysis when the evidence supported it. Gates also 
testified that if confirmed to the post he would use MacEachin’s analytic processes and 
standards as the model for all analysts (Gates, 1992b). 



41 


Douglas MacEachin, a career analyst at the CIA who served as Deputy Director 
for Intelligence from 1993 - 1995, reportedly told a colleague in 1994 that after reading a 
number of published intelligence assessments designed to support policymakers “roughly 
a third of the papers.. .had no discernible argumentation to bolster the credibility of 
intelligence judgments and another third suffered from flawed argumentation” (Davis, 
1999, p. xviii). He authored a monograph titled The Tradecraft of Analysis: Challenge 
and Change at the CIA in 1994 that addressed these concerns. Under MacEachin, 
uniform analytic procedures were established and disseminated to the workforce to 
provide “the standards against which products will be evaluated and as the measure of 
excellence for evaluating analysts’ professional performance” (MacEachin, 1994, p. 13). 
MacEachin also revamped the professional education curriculum among agency analysts 
and managers as part of this desire to define and institutionalize the analytic process. 

MacEachin’s Tradecraft of Analysis ultimately had as much impact on how the IC 
viewed analysis as Kent’s Strategic Intelligence. The difference in their works lies in 
their scope, in what portion of the black box of analysis they explored. Where Kent 
identified a wide ranging research process with analysts involved throughout, 
MacEachin’s monograph focused the discussion of tradecraft on step 7 of Kent’s model, 
presentation to the policy maker. To address the charges of politicization and his own 
observations of the quality of the analytic products, MacEachin also looked to improve 
the analytic output, identifying the attributes of an intelligence product. MacEachin had 
effectively operationalized analysis; the five principles of tradecraft he developed provide 
empirical indicators of what “good analysis” should contain (MacEachin, 1994, p. 2): 
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1. The credibility of the analysis is established through intelligence practices 
that clearly identify what is known, how it is known, and with what level 
of reliability. 

2. Analysts identify what is not known about an issue that could that could 
have important consequences. 

3. The “drivers” or “linchpins” (i.e., “factors which together would logically 
bring about a certain future” (MacEachin, 1994, p. 12)) that are likely to 
govern the outcome of dynamic situations are clearly identified. 

4. The analytic calculus underlying all conclusions and forecasts are evident. 

5. The uncertainties in any of the components of the analysis and 
implications of those uncertainties for alternative outcomes are made 
explicit. 

The focus of MacEachin’s tradecraft is to clearly communicate to the user of the 
intelligence product the information and methodology underlying the assessment. Or as 
Herbert (2006, p. 667) offered more than ten years later, “.. .the key issue in intelligence 
analysis [is] distinguishing genuine knowledge from mere belief.” This subtle, yet 
substantial change in scope can be seen when comparing Gates’ remarks to CIA 
employees in 1992 which identified analytic tradecraft to include everything from 
gathering data to publishing reports (Gates, 1992a); while MacEachin’s 1994 tradecraft 
standards were limited to analytic methodologies and improving product design. 

By 1997 MacEachin’s concept of tradecraft had been further developed within the 
analytic cadre of the CIA and A Compendium of Analytic Tradecraft Notes was 
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published. This manual contained 10 notes, or chapters, which expounded on the five 
principles of tradecraft he identified (Central Intelligence Agency, 1997). In 1999, a 
compendium of articles by Richard Heuer, Psychology of Intelligence Analysis, was 
published. Heuer’s focus was cognitive psychology and “how people process 
information and make judgments on incomplete and ambiguous information” (Heuer, 
1999, p. vii). While Heuer never used the term tradecraft in his articles, many were 
written before the term became associated with intelligence analysis, it is liberally used in 
the ex post introduction and foreword, and the book became a staple of tradecraft 
training. Psychology of Intelligence Analysis has become synonymous with the use of 
structured analytic techniques by analysts as a tool to identify and overcome unconscious 
cognitive shortcuts, and is still in use today as a tradecraft manual. 

Improving Analysis by Legislation: The IRTPA 
Following the 9/11 attacks and the subsequent failure to properly assess Iraq’s 
Weapons of Mass Destruction program in 2003, two intelligence failures exhibiting two 
completely different types of errors, (Betts, 2007) official investigations were conducted 
to determine their underlying causes. The National Commission on Terrorist Attacks 
Upon the United States, hereafter referred to as the 9/11 Commission, was operationally 
focused and tended toward examining the effects government organizations and 
operations in the failure. Still, the 9/11 Commission famously identified a “failure of 
imagination” within the IC as a key contributor to the successful 9/11 attacks (9/11 
Commission, 2004, pp. 339, 344 - 348). While identifying weak analysis as an issue, the 
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9/11 Commission recommendations were overwhelmingly focused on restructuring 
government organization, to include creating a National Intelligence Director. 

Conversely, the Commission on the Intelligence Capabilities of the United States 
Regarding Weapons of Mass Destruction, hereafter referred to as the WMD Commission 
(occasionally referred to in some literature as the Silberman-Robb Commission), focused 
almost exclusively on the intelligence community’s role in the failure concerning the 
assessment of the Iraqi WMD program. The WMD Commission report mentioned weak 
tradecraft as an issue several times, defining tradecraft as “.. .the way analysts think, 
research, evaluate evidence, write, and communicate...” (WMD Commission, 2005, p. 
12). Examining pre-war intelligence, the Commission’s findings echoed MacEachin’s 
criticisms of the late 1990’s when it stated that the finished intelligence products it 
reviewed were “.. .loosely reasoned, ill supported, and poorly communicated.. .we found 
too many analytic products that obscured how little the Intelligence Community actually 
knew [italics in original] about an issue and how much their conclusions rested on 
inference and assumptions” (WMD Commission, 2005, p. 12). Critical of the existing 
state of tradecraft in the IC, the Commission found the IC was “dead wrong in almost all 
of its pre-war judgments” constituting a “major intelligence failure” (WMD Commission, 
2005, p. 2). 

Of the 74 total recommendations the WMD Commission made, 16 specifically 
concerned intelligence analysis (The White House, 2005; WMD Commission, 2005). 

The wide ranging recommendations included improving the way the community 
identifies “key intelligence needs,” identified ways to improve data collection, identified 
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the need to develop tools to streamline the collection and evaluation of data, 
recommended specific analytic techniques to improve hypothesis generation, and 
recommended ways to improve presentation of the finished analysis. 

The Intelligence Reform and Terrorism Prevention Act of 2004 was drafted and 
passed following the completion of 9/11 report, concurrent with the WMD commission 
investigation, but seven months before that commission’s report was completed. The 
IRTPA “represents the most sweeping reform of our intelligence structures in more than 
50 years” (Collins, 2004, p. 302) and specifically addressed the need to improve analysis 
and analytic tradecraft (IRTPA, 2004, 118 STAT. 3650-3651, 3671-3672). The IRTPA 
legislated the standards of sound tradecraft as “based on all sources of available 
intelligence, properly describe the quality and reliability of underlying sources, properly 
caveat and express uncertainties or confidence in analytic judgments, properly distinguish 
between underlying intelligence and the assumptions and judgments of analysts, and 
incorporate where appropriate alternative analysis...” (IRTPA, 2004, 118 STAT. 3671). 
These standards of tradecraft codified in the IRTPA mirrored MacEachin’s five 
principles of tradecraft adopted ten years earlier focusing primarily on step 7 of Kent’s 
analytic process, presentation to policy makers. 

Following the passage of the IRTPA and the publication of the results of the 9/11 
and WMD commissions, a flurry of authors added to the discussion of analytic tradecraft. 
The majority of this post 9/11 material followed the lead of the IRTPA and took a more 
limited view of tradecraft. The most common theme identified the genesis and maturity 
of the IC in the Cold War era and used the Church Commission’s finding of the reliance 



46 


on a jigsaw approach to analysis as the primary cause of the failures and the logical focus 
of reform. Exemplified by Lahneman (2010), Moore (2011), and Hall and Citrenbaum 
(2010), all offer alternative analytic frameworks to break IC analysts out of the puzzle 
solving construct. 

Parker and Stern (2002) and Colby (2007) were representative of the perspective 
that while analytic frameworks may play a part in intelligence failure, the issue is much 
more complex and systemic. Parker and Stern (2002) examined the human psychology 
of analysts and strategic surprise while Colby (2007) identified the limits of analytic 
frameworks and empiricism to developing sound and useful intelligence products. The 
lack of information on the role of management in the analytic process was described as “a 
glaring hole” by Gentry (2016, p. 154) who went on to identify six analytic processes that 
he believed managers are directly involved in and seven corresponding “organizational 
pathologies” (p. 167). Going further, without focusing on analytic tradecraft explicitly, 
several authors (Betts, 1978, 2002, 2007; Clarke, 2008; Hedley, 2005; Phythian, 2006; 
Pillar, 2006a; 2006b, 2012) conclude that intelligence reform which focuses solely on the 
IC, and not on the intelligence / consumer / oversight melange is doomed to fail. As late 
as 2015 Lowenthal and Marks questioned whether “intelligence analysis may be as good 
as it can be expected to be” and stressed that “intelligence analysis is an intellectual 
activity, not a mechanical one where the proper formula or recipe will produce the 
desired recipe each time” (2015, pp. 664-665). 

In 2007 the Director of National Intelligence (DNI), a cabinet level office created 
by the IRTPA to address leadership and organizational issues in the IC, implemented 
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Intelligence Community Directive (ICD) 203, “Analytic Standards” (ODNI, 2015 a; Pigg, 
2009). This ICD was one in a series that the DNI developed to respond directly to 
IRTPA mandates, in this case to promulgate common analytic standards in the US IC. 

The ICD identified five IC analytic standards: objectivity; independence of political 
considerations; timeliness; based on all available sources of intelligence; exhibiting 
proper standards of analytic tradecraft (Director of National Intelligence, 2007; Pigg, 
2009). It then further defined eight standards of proper analytic tradecraft; a ninth 
tradecraft standard was added in a 2015 update to ICD 203 (ODNI, 2015a): 

1. Properly describes quality and reliability of underlying sources; 

2. Properly caveats and expresses uncertainties or confidence in analytic 
judgments; 

3. Properly distinguishes between underlying intelligence and analysts’ 
assumptions and judgments; 

4. Incorporates alternative analysis where appropriate; 

5. Relevance to US national security; 

6. Logical argumentation; 

7. Consistency, or highlights change; 

8. Accurate judgments, assessments. 

9. Incorporates effective visual information where appropriate. 
Unsurprisingly, the ICD used and built on the exact language from the legislation. 

MacEachin’s five principles of analytic tradecraft, first adopted in 1994 in response to the 
end of the Cold War and charges of politicization of analysis, (MacEachin 1994) and 
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already implemented prior to 9/11 and the prelude to Operation Iraqi Freedom, were 
carried forward into the IRTPA in 2004 and implemented verbatim in ICD 203 in 2007. 

By overlaying the nine DNI established tradecraft standards over Kent’s analytic 
process (see Figure 5) the focus on the output portion of the analytic black box model 
becomes evident. Eight of the nine clearly focus on clearly conveying information to the 
customer, only tradecraft standard 4 concerning the use of alternative analysis falls within 
the analytic process itself. The emphasis on the output reflects the problem associated 
with black boxing processes: without a clear understanding of what is happening, 


attempts to improve the process can only be made by addressing the inputs or outputs. 
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Figure 5. DNI analytic standards and Kent’s analytic process. 
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Summary and Conclusions 

Despite legislation and implementing directives, I have demonstrated in this 
literature review that the confusion underlying the understanding of what constitutes 
analytic tradecraft has yet to be definitively addressed. A 2008 study on intelligence 
analysis commissioned by the Air Force Research Laboratory found “a comprehensive 
review of the literature indicates that while much has been written, largely there has not 
been a progression of thinking relative to the core aspect and complexities of doing 
intelligence analysis...” (Mangio & Wilkinson, 2008, p. 1). Bruce and George (2008) 
found that the “body of scholarly writing on intelligence analysis remains.. .surprisingly 
thin” and that a review of 20 years of literature on intelligence analysis “yields meager 
results” (p. 3 - 4). In 2013, a researcher conducted a study of the topics were of articles 
published in the two leading peer reviewed and peer produced journals catering to 
intelligence professionals. Developing a coding scheme containing nine mutually 
exclusive categories he found that over the last 19 years only 6% of the published studies 
focused on analysis (Varouhakis, 2013). 

Examining the steady, if not worsening rates of intelligence failure since the 
creation of the IC, my assessment is that attempts to improve the analytic process by 
focusing on the inputs and outputs to the process have not had the desired results. 
Drawing on Kent’s admonition from over 60 years ago, without further research and 
academic discourse on what constitutes analysis and analytic tradecraft, the intelligence 
discipline, “.. .its method, its vocabulary, its body of doctrine, even its fundamental 
theory run the risk of never reaching full maturity” (Kent, 1955, p. 3). It is evident then, 
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that a gap exists on the analytic process and what skills constitute analytic tradecraft, and 
further research is on the subject is warranted. The following chapter addresses the 
methods this study used to begin that research. 
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Chapter 3: Research Method 

Introduction 

In the previous chapter I demonstrated how analytic tradecraft, which refers to the 
tools and techniques used by intelligence analysts to do their work, has been cited in 
literature, congressional hearings, and the press as a major reason for a series of 
intelligence failures. Beginning with the failure to accurately assess the decline of the 
Soviet Union, and magnified following the 9/11 attacks, the invasion of Iraq, and the 
Arab Spring, this perception of poor tradecraft among IC analysts resulted in mandated 
tradecraft standards in the latest intelligence reform legislation (9/11 Commission, 2004; 
WMD Commission, 2005). In reviewing literature, I found that analytic tradecraft is an 
understudied and poorly understood concept. Not surprisingly then, tradecraft reforms 
rooted in pre - 9/11 notions of analysis have not changed the pattern of intelligence 
failures (CBS, 2011; Koring, 2010; Lake & Lily, 2014; Liptak, 2016; Muhammed, 2014; 
Robbins, 2015; Sanger, 2014). The purpose of this study was to conduct research 
resulting in a better understanding of analytic tradecraft. 

In the previous chapters I defined the problem, the current state of scholarship on 
intelligence analysis, and the benefits that a study of analytic tradecraft can bring. In this 
chapter I describe the methodological underpinnings of my study. In it I discuss the 
research methodology and design; how data were collected, stored, and analyzed; the 
means I used to protect participants of this study; and my role in the research process. 
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Research Design and Rationale 

In order to better understand the tradecraft of intelligence analysis, the following 
research questions were addressed: 

RQ1. How does an intelligence analyst move along the process of analysis, from 
becoming aware of the need for an intelligence product to creating the finished 
product? The subquestions for RQ1 were the following: How is the analytic 
process learned; specifically, does it involve formal training, trial and error, 
intuition; and does the process vary within and between individuals (and, if so, 
what explains the variance). 

RQ2. What software, hardware, tools, people, knowledge, and so forth populate 
the analyst’s environment, and how does the analyst negotiate the environment? 
The subquestions for RQ2 were the following: Does the environment affect the 
analytic process, and can the analyst manipulate the environment to match needs, 
or does the environment determine the process. 

RQ3. What skills (technical, social, or other) are needed to navigate the analytic 
environment? 

The central phenomenon of the study was creation of an intelligence product by 
an analyst. In this study I built on the broad definitions of tradecraft such as the 
“practiced skill in a trade or art” (Johnston, 2005, p. 17-18) and “.. .the way analysts 
think, research, evaluate evidence, write, and communicate...” (WMD Commission, 
2005, p. 12). I determined that in order to best understand the tradecraft of analysts I 
should understand the process of analysis from the perspective of an analyst. 
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I designed this study with a qualitative narrative approach. A narrative is 
“understood as a spoken or written text giving an account of an event/action or series of 
events/actions, chronologically connected” (Czamiawska, 2004, p. 17). Narratives are “a 
basic human strategy for coming to terms with time, process, and change...” (Herman, 
2009, p. 2); and are “a practical solution to a fundamental problem in life, creating 
reasonable order out of experience” (Moen, 2006, p. 2). Narrative researchers ask the 
participant to relate the experience which is the focus of the study in the form of a story, 
to put a beginning, middle, and end to a sequence of related events (Josselson, 2010; 
Rankin, 2002). Conducting narrative research by analyzing and restorying the detailed 
accounts of individuals to understand specific experiences in context is a recognized 
approach to studying the role and actions of the individual in organizations (Czarniawska, 
1998, pp. 2-8). 

Intelligence analysis, whether viewed as a process that spans Kent’s seven steps 
or MacEachin’s more limited perspective (Kent, 1949, pp. 157 - 158; MacEachin, 1994, 
p. 2) is still a sequence of events that are acted out and experienced by individuals within 
the context of their environment. As Latour stated in a discussion of ANT, “actors know 
what they do and we have to learn from them not only what they do, but how and why 
they do it” (1999, p. 18). In order for this research effort to obtain a better understanding 
of how analysts do their job and the skills, techniques, and tools they use to do it, 
following their experience of creating an intelligence product is a methodologically 


sound approach. 
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A researcher following a narrative approach can be guided by a theoretical or 
conceptual framework (Josselson, 2010). As the participants’ stories are deconstructed 
and restoried to obtain a clearer understanding of the skills of an analyst, it was important 
for me as the researcher not to attempt to fit those stories into a preconceived notion of 
tradecraft. By adhering to the principles of ANT (i.e., agnosticism, generalized 
symmetry, and free association (Callon, 1986; Crawford, 2004; Delukie, 2009)) with the 
rich description found in the narratives, I was able to provide a more complete 
description of activities of the analyst in the heterogeneous network of the IC and the 
transformation of information and knowledge into an intelligence product. Using ANT in 
conjunction with a narrative approach allowed me to use analysts’ descriptions of the 
sequences and process involved in their work to develop a more complete understanding 
of analytic tradecraft. 

Methodology 

Participant Selection 

The population of possible subjects was the pool of all-source analysts currently 
serving in the U.S. IC. While reliable estimates of the number of personnel in the IC are 
difficult to come by, they are estimated to be in the hundreds of thousands when 
including civilians, military, and contractors (Priest & Arkin, 2010). An unknown 
fraction of these are analysts (ODNI, 2006b). Analysts in the U.S. IC, including 
civilians, military personnel, and contractors, can be found in locations as diverse as 
active war zones, federal law enforcement field offices, Department of Homeland 
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Security fusion centers, military service intelligence centers, and the White House (DNI, 
2007b, pp. 9-11). 

A narrative however, does not rely on a large population sample; instead it 
explores in depth the experiences of an individual or small sample of subjects. I 
conducted interviews with seven analysts which provided the narratives used as the basis 
in this study. Saturation in a narrative study is not achieved by drawing from a large 
sample size, but by fully eliciting the details of the experience which is the focus of the 
study. I found that the data provided by these interviews provided the saturation needed 
for the study, not requiring additional interviews or alternate data collection methods. 

I used purposeful case sampling as described by Miles and Huberman (1994, pp. 
27-29) with the intent of generating a wide variation of experiences; providing me more 
confidence is addressing and describing the representative environment of analysts. I 
used an interview guide to conduct my interviews with analysts currently working in IC 
elements, ensuring that all participants were queried on the same elements of their actor- 
network environment while maintaining the flexibility and freedom for the analysts to use 
their own thoughts and words to define that environment. I searched for available 
statistics and demographic data on the analytic workforce from the Office of the Director 
of National Intelligence in order to identify attributes associated with analysts (e.g., years 
working as an analyst, number of analysts assigned to IC elements) and to guide subject 
recruitment for the study, however none were openly available. The criterion I used to 
obtain the variety in participants were years of experience as an analyst, analytic 
specialty, and agency assigned. 
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Posts were placed in selected electronic bulletin boards on internal IC analytic 
collaborative networks advertising for individuals willing to participate in the study. The 
ability to see and respond to the solicitation, placed on the internal network, sufficed to 
substantiate the respondent’s position within the IC. An initial interview, designed to 
further describe the study and the role of the participant, also served to verify the 
potential subject’s individual background and qualifications for study selection. 
Instrumentation 

I collected data using a semi-structured interview process designed to draw on the 
experiences of the subjects as they described the analytic process. I maintained the 
collaborative nature of the interview by allowing the participant freedom to recount their 
story as they wished, using the interview guide as a tool to ensure that no major topic 
areas were overlooked. Questions in the interview guide generally followed Kent’s 
(1949) analytic process framed against the three research questions. A sampling of 
questions is found in Appendix A. 

I used publicly available documents, primarily vendor provided information on 
specific systems or tools identified by participants, as needed to provide context to the 
analysts’ narratives. With ANT as the theoretical guide for this narrative study, the 
words of the subjects themselves were the most important element of the data collected. 
None of the participants offered any drawings or sketches to supplement their narrative, 
nor did any offer any particular document or presentation as crucial in understanding 


their story. 
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As I developed my research methodology I incorporated the concept of 
sufficiency in qualitative research. As sufficiency in a narrative is not determined by 
volume of data but through the depth and detail of the story that the participant provides 
the researcher, by creating the environment where the participants had the time and 
intellectual space to completely describe their experiences in the analytic process I 
allowed for sufficient detail to be provided. Eliciting the participants’ description of the 
analytic process in their own words and using ANT as the conceptual framework I 
established the means necessary for a sufficient and clearer understanding of analytic 
tradecraft. 

Validity, or the correctness of the interpretation of tradecraft gained during the 
analysis, cannot be guaranteed by method or instrument selection (Maxwell, 2005). In 
Creswell (2007) I identified three areas to be cognizant of in order to provide content 
validity: the ability of the subject’s experiences to address the research questions; 
recognizing my own biases and ensuring that my voice was not substituted for the 
participant’s; and the accurate representation of the narrative as confirmed by the subject, 
official records, or other accounts. I address all three of these elements in the following 
sections. 

Data Collection and Analysis 

Procedures for Recruitment, Participation, and Data Collection 

I recruited participants for the study through posts made to an analytic 
collaborative web sites. The post identified that volunteers were needed for a doctoral 
study of analytic tradecraft and that participation included an in-depth interview on their 
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analytic process. There were no incentives offered for participation. In order to facilitate 
the sampling strategy I conducted interviews with volunteers in the U.S. and Europe in 
person, via skype, and phone. All necessary screening of the participants to determine 
their eligibility was done by phone and email. 

The participants were all located away from their immediate work environment, 
when they participated in the interviews, which provided both privacy and a sense of 
ease, and further reinforced the unclassified nature of the research. None of the 
participants objected to the interview being recorded, the recordings were supplemented 
with my field notes. As the researcher, I conducted all transcriptions of the recordings. I 
provided participants transcripts of their interviews as a member check to validate their 
collected experiences and provide the opportunity to correct or elaborate on any 
statements. 

At the completion of the study, subjects were notified that the dissertation was 
complete and they were thanked for their participation. Electronic copies of the 
completed dissertation were provided to them. No other follow up procedures or contact 
was necessary or contemplated. 

Data Analysis Plan 

I coded and analyzed the data, in the form of the stories told by the subjects 
transcribed into text, guided by the tenets of ANT. To that end, I did not create list of 
codes prior to the analysis phase so as not to project my vocabulary and pre-conceptions 
onto the participants’ voices (Latour, 2005). Instead, as researcher, my analytic focus 
was on the transformations occurring within the black box of analysis and the 
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identification of actors in the participant’s environment (Latour, 2005; Maxwell, 2005). 
The skills needed to successfully operate in this actor-network of analysis represent 
analytic tradecraft as identified by the analysts. Data identified as dramatically 
discrepant, or data that resisted coding, was treated as an impetus to review the initial 
study assumptions. 

I used MAXQDA 12 data analysis software to automate my coding and 
organization of the data. Software has been used in previous research utilizing semi- 
structured interviews as the basis for data collection and the ANT framework (Brandt, 
Dalum, & Thomsen, 2013; Papadopoulos, Radnor, & Merali, 2009; Valentine, 2007) and 
yielded sound results. The use of MAXQDA 12 also provided export tools allowing me 
to use other office automation for analysis and presentation. 

Role of the Researcher 

The dual roles of a researcher, both the instrument that collects the data as well as 
the analyst that interprets it, required that I approach the study with the explicit intent of 
building on the strengths of my personal experiences while developing a neutral and 
authentic representation of the findings. As the researcher of this study, I bring over 30 
years of continuous service in the U.S. IC through the Department of Defense. First 
commissioned in the Army Intelligence Corps, I have also served in the IC as a defense 
contractor and a government civilian. I have served as analyst, collection asset, staff 
officer, and intelligence program Inspector General at levels of responsibility from 
tactical to national. The majority of this time, over 15 years, has been as a theater - 
national level intelligence analyst pursuant to my designation as a strategic intelligence 
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analyst following the conferral of a Master of Science in Strategic Intelligence from the 
National Intelligence University, and assignment to Functional Area 34, Strategic 
Intelligence Officer. During the time I conducted this study, I was a full time employee 
of an IC element, working as an intelligence analyst. 

The use of the narrative approach guided by ANT assisted me in reducing the 
effect of my personal biases formed by my experiences. The narrative approach focused 
my efforts on accurately capturing the experiences of the subject, while the ANT 
framework mandated I analyze the data using the subjects’ own words without imposing 
a pre-determined order. This approach allowed me to capitalize on personal experiences 
when working with the study participants, while reducing the effect of those experiences 
on interpreting the data. 

Participants knew me primarily through my role as a researcher and student, 
however the use of i-Space as a recruitment tool inevitably disclosed my position as an 
analyst in the IC. To reduce the effect of any bias based on agency practices, no 
participant was chosen from my current IC assignment. My role of student and 
researcher also reinforced the need to keep all discussions unclassified. As researcher, I 
emphasized the academic nature of the study and that the narrative is about the journey of 
creating an intelligence product, not on the specifics of the intelligence itself. 

Trustworthiness 

Trustworthiness is an overarching expression of the quality of a qualitative 
research endeavor, encompassing the elements of transferability, credibility, 
dependability, and confirmability (Given & Saumure, 2008). As researcher, it was my 



61 

responsibility to develop a strategy that considered how these elements would be 
evaluated and consider ways to mitigate threats to trustworthiness early on. This section 
discusses how each of these elements were considered in this study. 

Transferability, analogous to generalizability or external validity in quantitative 
studies, can best be addressed by a well thought out sampling strategy and presenting the 
findings in the proper context (Jensen, 2008d). As described earlier, purposeful case 
sampling was chosen due to the low level of existing knowledge of tradecraft in the IC as 
a means to best address community wide gaps in knowledge. This technique assisted in 
providing findings which are the most transferable across the IC. However, the resulting 
context must be considered, especially given the wide variety of possible environments 
that IC analysts may find themselves also discussed previously and the wide latitude 
individual agencies have in setting policy and procedures. The tradecraft of an analyst 
working in a national agency in Washington DC may be analogous to those in a regional 
fusion center or deployed to a forward headquarters in Afghanistan, however tools, time 
available, and organizational policies are very likely different. As I will address in the 
following chapter, in order to maximize transferability actors in the analytic process were 
aggregated to a common level of understanding. 

The research questions were designed to elicit general observations and 
experiences of tradecraft from the analyst. In addition to avoiding potentially classified 
discussions it also served to improve transferability of the findings. By not focusing on a 
particular product or event which the analyst participated in the likelihood that the 
findings on tradecraft can be generalized across the wider IC was improved. 
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Credibility in a qualitative study has been defined as “the methodological 
procedures and sources used to establish a high level of harmony between the 
participants’ expressions and the researcher’s interpretations of them” (Jensen, 2008b). 
To address credibility I reviewed the elements of the study that concern ensuring the 
correct population, methodology, and data collection techniques were chosen to address 
the research question. My approach of selecting currently working analysts was used to 
provide a solid insight into tradecraft, my use of the narrative approach guided by ANT 
was designed to elicit a more complete understanding of analytic tradecraft, and my use 
of the semi-structured interview provided the opportunity for the subject to provide the 
most complete and thorough account of the transformation from information to 
intelligence. 

Dependability is closely aligned with the quantitative concept of reliability, or the 
ability to replicate a measurement. In qualitative studies however it is never possible to 
exactly replicate the situation and environment in which the data was collected, so as 
researcher I strove to accurately provide an account of the steps of the process 
(Golafshani, 2003; Jensen, 2008c). My use of MAXQDA 12 as analytic software also 
served to provide an accurate and verifiable account of the analytic process. 

Additionally, given the narrative approach taken, member checks were conducted with 
the subjects to ensure that the accounts of their experiences were accurately captured. 

Confirmability “refers to the degree to which the results could be confirmed or 
corroborated by others” (Trochim, 2006). In order to do so, I was transparent in the 
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reporting of how the data was collected, coded, and analyzed. Additionally MAXQDA 
12 is again useful in providing an automated data trail (Jensen, 2008a). 

Ethics and Confidentiality 

I obtained Walden University’s Institutional Review Board (IRB) approval, 
number 04-07-15-0202252, prior to any effort to recruit or contact potential subjects. 

The population for this study does not represent a vulnerable population and there was no 
professional coercion to participate. All identifying information was removed from the 
data before entry into MAXQDA 12 and the participants were identified by a confidential 
coding system. Participants signed a participation agreement and were advised of their 
rights as subjects in the study. No incentives or compensation was offered for subject 
participation. As researcher, I transcribed all interview recordings myself. At the 
completion of the study, all hardcopy and electronic data was consolidated and will be 
stored in a secure container for 5 years, at which point a determination will be made 
whether further retention is needed. 

Summary 

In this chapter I described the methodological underpinning of the study. I 
discussed the research methodology and design; how data was collected, stored, and 
analyzed; means used to protect the subjects of this study; and the role of the researcher. 

I also provided an understanding of the process by which the research questions were 
addressed in this study. This chapter provided the background for the following chapter, 
in which I address the results of the study. 
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Chapter 4: Results 

Introduction 

My purpose in conducting this research was to develop a better understanding of 
analytic tradecraft. I designed the study with a narrative approach using actor-network 
theory as the conceptual framework (Latour 2005). In order to understand how analysts 
do their job, I posed the following research questions and subquestions: 

RQ1. How does an intelligence analyst move along the process of analysis, from 
becoming aware of the need for an intelligence product to creating the finished 
product? The subquestions for RQ1 were the following: How is the analytic 
process learned; specifically, does it involve formal training, trial and error, 
intuition; and does the process vary within and between individuals (and, if so, 
what explains the variance). 

RQ2. What software, hardware, tools, people, knowledge, and so forth populate 
the analyst’s environment, and how does the analyst negotiate the environment? 
The subquestions for RQ2 were the following: Does the environment affect the 
analytic process, and can the analyst manipulate the environment to match needs, 
or does the environment determine the process. 

RQ3. What skills (technical, social, or other) are needed to navigate the analytic 
environment? 

This chapter is divided into seven sections. Following this introduction, I discuss 
specifics of how the data were collected. I then address data analysis. My discussion 
includes a brief review of my guiding conceptual framework, ANT. In the fourth section, 
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I provide the results of my investigation. This section first includes a description of the 
findings relying on the participants’ narratives, which is followed by a graphical portrayal 
of those results. After presenting the results, I offer my conclusions for the study. Then, 

I evaluate the trustworthiness of the study’s results. The final section includes a chapter 
summary. 

Data Collection 

Recruiting 

In accordance with guidelines in my IRB approval letter, I initially recruited 
participants via a collaborative website catering to those with an interest in analytic 
tradecraft and analytic methodologies. The website, Analytic Methods Forum, operates 
on the Joint Worldwide Intelligence Communication System (JWICS), a U.S. 
government secure network available only to individuals with security clearances and 
access to classified information. Approval to post on that website was obtained after 
contacting the site owners and satisfying their concerns regarding the non-proprietary 
nature of the research effort. At the time of recruitment, the site had approximately 300 
members listed, not all of whom were active participants. 

After 9 months, seven individuals had responded to the invitation to participate, 
and only four of those respondents met the sampling strategy and agreed to be 
interviewed. Realizing that the pool of recruits available through that site was likely 
exhausted, I requested a modification to my IRB approval, which was granted. The 
modification allowed for an additional recruiting post on a membership restricted online 
forum known as INTELST, which has been described as “a Pentagon hosted email 
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discussion group for discussions of intelligence and military intelligence matters” 
(Holden, 2015, p. 1). At the time that I made the recruiting post, the forum had over 
4,000 members across the entire U.S. IC, international partners, academia and business 
Holden, 2015, p. 2). As a result of this second recruiting post, I received an additional 
seven responses and the remaining three participants were selected. 

Four of the seven participants were employed as intelligence analysts by the 
Department of Defense at the time of their interviews. They had experience ranging from 
combat tours with tactical units in Iraq and Afghanistan to analytic assignments with 
combatant commands. The remaining three participants were employed as intelligence 
analysts for national level non-DoD IC agencies. Analytic experience levels of the 
participants ranged from 5 to 20 years, with the average being slightly under 12 years. 

As discussed in Chapter 3,1 did not consider gender and age as selection factors. 
Interviews 

I conducted seven interviews from June 2015 through March 2016. Four 
interviews were conducted over the phone while two were conducted in person and one 
via Skype (see Appendix B). All interviews were recorded with the permission of the 
participants, were transcribed by me, and member checked by the participants. I 
conducted all the interviews away from the participants work environment, which served 
to reinforce the unclassified nature of the study. I used an interview guide (see Appendix 
A) which used open ended questions to provide the participants intellectual space to 
explain the analytic process in their own words. 
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The member checked interviews were saved as Microsoft Word files and 
identified by the dates they were conducted. I removed all potentially identifying 
information, such as personal names, agency names, and gender references, during the 
transcription process. These redacted, member checked digital files were then uploaded 
into MAXQDA 12. 

Data Analysis 

Implementing the Analytic Framework 

As I described in Chapter 2, ANT was originally developed as a means of 
understanding how the scientific community created knowledge (Law, 1992). As it was 
developed and applied to other disciplines, it became a tool for thinking about and 
describing social actions. ANT became an approach to explore and map the social 
environment where transformations take place, to identify the heterogeneous actors in a 
society, their interactions and outcomes (Latour, 2005, pp. 64, 179-180). At this point, I 
think it is useful to review some key concepts of ANT and the lexicon associated with it, 
specifically as they apply in this study, in order to facilitate the ensuing discussion. 

In ANT, an actor is an entity that is the source of an action in the course of a 
transformative process; actors cause things to happen. (Latour, 1996; Latour, 2005, p. 
107; Venturini, 2010, p. 266) In this study, actors, also referred to as actants to avoid the 
assumption of human agency, are those things that analysts identify as having an effect 
on the analytic process, those entities (including themselves) causing other things to 
happen in the process. (Latour, 1996; Latour, 2005, p. 76, 143). For example, an agency 
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policy that restricts the length of an intelligence product to one page causes the analyst to 
create a product conforming to that limit. 

A trace of the transformations that occurs among actors is the actor-network. As 
Latour states, a network “is a tool to help describe something, not what is being 
described.” (Latour, 2005, p. 23, 108, 131) As I develop a more comprehensive 
understanding of the analytic process during the course of this study, the actor-network 
will be the trace of activity inside the analytic black box. 

A transformation is the process that occurs due to the activities of actors, it is a 
description of the changes that have taken place. (Latour, 1996; Latour, 2005, p. 39, 64, 
107 - 108). To answer the research questions in this study, I am interested in how the 
inputs of the analytic process, requirements and reporting, are transformed into 
intelligence products and additional collection requirements. This transformation is what 
the black box of the analytic process obscures. 

The process by which actors in transformations enlist other actors to support their 
position in the process is known as translation. In a social environment some actors are 
able to translate other actors’ positions into an agreed upon group position, thereby 
stabilizing the transformative process. (Callon and Latour, 1981; Latour, 2005, p. 108; 
Law, 2007; Toennesen, Molloy, & Jacobs, 2006). In ANT studies there is no a priori 
assumption of which actor may be dominant during a transformation, based on rank or 
position for example, instead all actors are assumed equal until the relationships within 


the transformation can be determined. 



69 


Coding and Analysis 

I began to inductively code the interview transcripts after the final interview was 
completed. As outlined in Miles and Huberman, (1994, pp. 57-78) following several 
readings of the transcripts I developed a series of descriptive codes to identify a wide 
range of themes. I initially identified 16 codes, which were then retrospectively 
interpreted into the four broader categories of “role of the analyst,” “stages of the analytic 
process,” “the process of analysis,” and “tools used in analysis.” This coding schema 
provided the initial understanding of how analysts understood the process for which they 
were responsible, the environment in which they operated, and the actors that populate it 
(see Appendix C). During this initial coding pass I did not identify any discrepant cases. 

Once I had conducted the initial coding and gained an understanding of the 
participants’ perspective of the analytic process, my next step was to gain a more cogent 
understanding of the actors involved. A critical aspect in my data analysis was to first 
determine the level of actor-network that was most appropriate to be traced. Within 
ANT, all networks are comprised of actants, and all actants are in turn networks within 
themselves (Law, 2007; Nyakuengama, 2014). Without understanding the context of the 
social environment an actor-network can be over-described or under-described; taken to 
the extremes actants can be deconstructed to the cellular level or assembled into a series 
of black boxes that do not provide insight into the transformations. For the purpose of 
this study and to address the research questions, I chose to trace the actor-network at the 
level meaningful to the analyst as articulated in the participants’ narratives and identified 
in the first coding pass. Given the intended diversity of the participant population as 
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regards to IC agency assigned and experience level, I aggregated similarly described 
actants as necessary into more generalizable categories (e.g., management, office 
automation). 

At this point, having earlier identified the inputs to the analytic process, 
specifically requirements, and reporting (see Figure 3), I used the four broad coding 
categories as my roadmap to further explore the narratives to identify transformation 
points. I used several techniques to identify the points of transformation and the actants 
involved within the analytic process. Using MAXQDA 12 I exported the codes and 
associated text into Microsoft Excel documents, allowing me to copy, sort, and group 
phrases. I also used markers and easel pad paper to sketch out emerging themes; 
transferring those rough concepts to Microsoft PowerPoint once they were sufficiently 
developed to graphically portray the groupings and assign icons to them to assist in 
visualizing the data. I experimented with several different approaches in PowerPoint to 
portray the actor-network, balancing the need for readability while preserving the desire 
to portray complexity. 

Results 

I identified five discrete transformation points in the narratives, where actants 
working together resulted in a change of state to the inputs of the analytic process. I 
named these points: Ti, Requirements => Questions; T2, Reporting => Information; T3, 
Information => Evaluated Intelligence; T4, Evaluated Intelligence => Answers; T5, 
Answers => Products. Having identified these five points, I revisited the narratives with 
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a focus on detailing the transformations and the actors involved, and conducted additional 
coding passes where I applied the codes “transformation” one through five. 

I will present the results of this analysis in two parts. First, I will provide extracts 
of the participants’ narratives describing their first-hand experiences of the analytic 
process. In the second part I will provide the resultant actor-network trace of the analytic 
process and the actants involved. 

A Narrative Description of the Analytic Process 

Ti: The Transformation of Requirements into Questions. The first 
transformation the participants described was that of requirements into questions. The 
analysts related that rarely are requirements in a form that they can answer directly, and 
that a clear and common understanding among all actors involved in the process of the 
question being addressed is crucial. As Analyst 4, an analyst at a non-DoD IC agency 
states “you have to know what you are looking for or you will derail, go off on tangents.” 

Requirements analysts receive can range from standing, long-term broad 
information needs about capabilities and intentions, to very specific and time sensitive 
questions concerning developing situations (ODNI, 2014, pp. 4-6). While the 
participants identified the need to transform the requirements, the specifics of how that 
was done varied among the participants. The participants’ work environment dictated the 
actants involved and the relationships between the actants. 

Analyst 5, an analyst for a non-DoD national agency, described how requirements 
are transformed into questions as part of a deliberative process within the agency for 
longer term, planned production. 
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The big production pieces, we are very much on a production cycle, it’s a fiscal 
year thing... .And so our team here will brainstorm a set of proposals, try and 
demonstrate how they respond to KIQ [key intelligence questions] sort of things, 
and how they also respond to some special capability that we have here. Because 
there are all sorts of KIQ that we don’t have anything to do with. There has to be 
an overlap between what they want and what we can provide. So we’ll put 
together a short list of half a dozen or so, and then they’ll get fleshed out to a few 
paragraphs, then the local program, project officers, here will stack them and 
prioritize them based on their opinion of what’s most sellable and what’s 
important. 

Analyst 2, an analyst at an operational level combatant command in the DoD 
responsible for a regional portfolio of countries and a customer set ranging from national 
level policy makers to tactical level subordinate units, describes the process of 
determining questions from multiple customers that may not always understand 
themselves what they are looking for. 

The first question is of course, what is the question. So the first step is to know 
what your customer requirements are, what are they trying to answer, what are 
their problems, even more generally what do they think the issues are... .So I’m 
working for an activity now that’s working for different entities, sometimes the 
requirements are competing... .What’s the real question, what are they asking, we 
get a lot of bad questions, a lot of bad RFIs, [request for information] we use that 
rephrasing technique to understand what do they really want to know. 
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Multiple customers with different perspectives on an intelligence problem is not 
just an issue for analysts working at the operational and national level. Analyst 7, an 
active duty military intelligence officer with two combat tours describes the tactical 
experience on the battlefield, and the need to understand and translate sometimes 
unspoken requirements. 

The battalion commander is going to have a different requirement than the 
lieutenant that is going to lead the patrol. So the commander is going to be more 
interested in things like the overarching statistics, more the big picture kind of 
thing. The patrol leader is going to be looking for things like, in this one tiny area 
what exactly am I looking for, who are the specific individuals that might be of 
interest here, what are the very particular threats that are here and maybe not 
somewhere else... .he cares about what’s going to happen to him in the next six 
hours, he really doesn’t care about what USAID [US Agency of International 
Development] thinks, or the State Department thinks. 

Analyst 1, an analyst with over 20 years’ experience, currently working in a non- 
DoD IC agency with duties as an analyst on an interagency committee, discussed the 
fortunate position of being able to maintain a close working relationship with the primary 
customer. 

.. .it makes it easy for me to identify intelligence questions.. .because we typically 
have only one or two targets to evaluate... .once we produce our initial 
assessments we will often get feedback from the policy makers and that’s what 
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unique about my position now... .1 meet with the policy makers every week... .1 
guess our intelligence loop is very tight which helps us greatly. 

This perspective, of maintaining an awareness of a primary customer’s needs, 
whether articulated formally or not, was echoed by Analyst 6, an active duty military 
intelligence officer and analyst at another combatant command responsible for regional 
issues. Analyst 6 described how it is important to not just have an understanding of 
standing requirements, but a broader understanding of the customer’s mission and 
objectives. This insight allows an analyst to be able to identify requirements as they 
might emerge and provide warning intelligence for a problem that might not yet exist. 

.. .we work with partner countries... .doing military capacity building, that sort of 
thing. In order for any military to build capacity they have to have a national 
budget, so they can buy their equipment, resources, and training... .it happens we 
were reading an article about bananas.. .so 99% of the world’s bananas are one 
type and they come from Central America, it’s their major legitimate export. But 
there’s this fungus, so that crop is at risk, the entire crop is at risk of being non- 
viable within some number of years... .and realize that thing in the future could 
impact his current mission today even though his mission is military capacity 
building. 

From the analysts accounts we see that at any one time there are a number of 
forces at work transforming the requirements into questions. Whether the requirements 
come in the form of national level KIQ, operational level RFI, an understanding of the 
mission, or from a conversation with a patrol leader, that need is transformed into a 
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question or series of questions the analyst will attempt to answer. These questions 
provide the basis for the next step of the process. 

T2: The Transformation of Reporting into Information. The second point of 
transformation I identified occurs when analysts sort through the myriad types and 
volume of data available to them to create information relevant to their portfolio and task. 
As described in the narratives, analysts throughout the IC have any number of search 
tools available to them on classified and unclassified computer networks, and depending 
on which agency they are in within the IC they may also have access to physical 
resources such as libraries and the ability to talk to subject matter experts. It is this 
information, colloquially known as reporting, that is used in the production of finished 
intelligence. I determined in my conversations with the analyst participants that this 
wealth of data sources only represents potential knowledge; it is up to the analyst to find 
the useful information. 

The participants identified a number of basic tasks analysts were expected to be 
able to accomplish. This ranged the gamut from understanding Boolean logic in order to 
build search queries, being able to identify and parse relevant information from any 
number of data sources and then sort through and organize that data into personal or 
community accessible databases, to populating automated analytic tools. Analysts also 
described that it was important to be f a miliar with the people comprising the community 
of interest involved with their analytic portfolio as an information resource. During the 
interviews, the analysts describe that at this point they are not processing the reporting 
into meaningful segments of information, but instead are searching for, collecting, 
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culling, and then storing what may be relevant from the much larger universe of 
information available to them. 

Knowing where to look, where to begin to cast the net for information that may or 
may even be useful to analytic problem is a skill usually learned by working with other 
analysts and simple exposure to assets during the course of working. As Analyst 4 
describes, there is no simple solution or shortcut to knowing what is available to the 
analyst. 

So very critical to having good tradecraft is knowing where you can get all your 
information from. You know I think this is the part where experience and s ki ll 
bring up the caliber of the tradecraft because as a new analyst you’re just not 
going to have that.. .it’s just not available, no single agency or analyst is given a 
list of every single database that there is. It’s just not possible and you have to be 
willing to move out and meet people and talk to people and ask those questions, 
that’s important. 

When describing the role of the analyst in being aware of information from 
diverse sources Analyst 3, a civilian analyst with experience in two DoD analytic centers 
currently working at a combatant command, describes an example of the wealth of 
information available from academic sources and the need to integrate it with clandestine 
collection to get to a complete picture needed to begin the process of answering the 
question. 

A lot of questions that we work to write assessments on, a lot of times that answer 
has already been provided. Specifically if you read the news for a while foreign 
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fighters were all over the news. Well there are people that have written full 
dissertations on foreign fighters, and so I guess it is a skill to be able to go into 
academia and pull out the main findings from that and to translate it in a way that 
useful in an intelligence product....and the analyst, it seems like they’re in a 
unique position to be able to do that. Because not only are you fully grasping and 
bringing in academic literature but you have the access to compare it against, I 
guess the intelligence. It’s funny because when people talk about intelligence, 
and it’s like this super-secret information, and it’s got to be better than everyone 
else’s information, but actually it’s just one more piece in a mosaic of information 
that you have to consider. Because past information has its place, academic 
information has its place, intelligence information has its place, but no source of 
information is enough by itself. 

Analysts related that knowing where to look is only the start of the 
transformation. The process of extracting information, whether from running queries 
against an IC database or debriefing a patrol and capturing their firsthand knowledge of 
the operating environment, can involve any number of disparate actors. Analyst 7 talks 
about the use of a specific tool to overcome the friction of competing requirements in a 
combat environment to collect information for later analysis. 

The TIGR 3 system is actually really helpful as well, don’t know if you’re familiar 
with that. It’s a lot like Google Earth, but it’s interactive.. .which makes things 

3 Tactical Ground Reporting System (TIGR) is described by the developer, General Dynamics, as an 
information centric solution using a Google ® Earth like interface that empowers users at the tactical level 
to collect, share, and analyze data. See https://gdmissionsystems.com/c4isr/tigr/ 
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like patrol debriefs a lot easier. The platoon can just go in there and drop points 
on a map and I can pull it up and refresh it almost instantly and I can say ‘I can 
see a pattern forming here because I’m getting information from 1 st Platoon and 
3 rd Platoon’ and I don’t have to hunt down that Lieutenant that has a million other 
things to worry about and try to drag intel out of him right after a mission when 
the last thing he wants to do is to sit there and talk, he just wants to chuck his gear 
and take a shower. 

Away from the tactical environment, many searches for information begin in IC 
databases. As described by the participants, this assumes a knowledge of query writing, 
understanding the types of information available in particular databases, and background 
knowledge on the question itself. Analyst 4 described a series of repeated drill downs, 
balancing the need to identify relevant information while limiting the results to a 
manageable number. 

I would start with the most general database that would have all reporting... .1 
would want to develop a query. Not too specific, but specific enough to include 
as many key words attached as ors. You know, there would be groups of this set 
of key words - this or this or this - and then and - this or this. Because there are 
all sorts of synonyms or other ways of saying it, so you want to capture as much 
specific information to your question. So you want to limit the information, but 
capture all the different variants of how this information might be picked up based 
on key word searches... .for example, if you’re looking at a specific technology 
there are tons of words out there for components of a technology. One report 
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might be talking about a specific component but if you’re interested in the 
technology at large you can’t just search on the big name of the technology. You 
have to understand all the different critical components of it so that you can 
search on all of those. And all of them will make up this big piece. 

While in general agreement, at the same time, Analyst 2 cautions against 
becoming complacent with queries or databases, and alludes to the tensions analysts face 
when completing a seemingly mundane task such as running database queries to achieve 
a production requirement. 

My inclination was always to go and read more, go research more, and I just think 
there’s a lot of information now a days, intelligence information that we’re 
collecting. The whole problem of having more information than we can process. 
And so what you need from analysts, you need analysts who are out there, who 
are willing to jump from information to information instead of the more sort of 
insular hedgehog types that I probably tend to be, and probably still am 
unfortunately... .Like for example our analysts tend to find a favorite INT, or a 
favorite search query or something, and often times that gets them results that will 
meet their production requirement... .And I would constantly try to caution them 
against doing that. 

Finding information in IC databases can involve more than writing a query, and 
involve more actors than a solitary analyst, a keyboard, a Boolean logic cheat sheet, and 
access to a database, especially when dealing with a complex problem or one a particular 
analyst may not be especially comfortable with. As Analyst 4 relates 



Maybe if you’re not shy you would ask for support from your peers to take a look 
and maybe there are some other key words that you may have left out. One 
advantage we have here.. .we have a senior scientist attached to our division and 
he’s that person that we can go to and say what does this component mean, what 
are other words for this, and he can typically help us with that kind of stuff. 

Once reporting is extracted as data, again from any number of myriad sources 
running the gamut from IC specific intelligence databases, professional or academic 
journals, news reports, or from patrol debriefs, the analyst must essentially triage that 
data and store what has been harvested so it can be retrieved and fully processed at a later 
point. Generally, how that is completed is an individual decision based on preferences 
and tools available. As Analyst 3 relates 

Gosh, this is going to be, this is going to sound really rudimentary, but I still keep 
notes with paper and pen, just because there is just something, I can just process 
information better if I’m taking notes by hand. Sometimes I will catalog reports 
that are of interest to certain products in just plain Excel. 

Writing the information down by hand, or handling paper copies of the data 
makes was a preferred method of data collection and storage among most of the 
participants. As Analyst 1 said 

I’m a paper person, I guess I’m old fashioned. Typically when I’m scanning 
through reporting and I find something interesting the first thing I’ll do is print it. 
For me, I know it’s a waste of paper, but for me it does something different to 
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actually have it in your hand, scribble on it, underline things, it does something 
for me that sticks in my brain differently than if I’m just reading it on a screen. 
Others, such as Analyst 5, choose to use digital storage methods as a means to 
store and catalog the collected data. 

Then often I find myself summarizing all the intel in an Excel spreadsheet. Again 
it can be chronologically, but then I can sort it any which way I want. And this is 
nothing terribly extravagant, I just do it myself, by serial number, date of 
publication, date of information, particular names, so I can get it all in there and 
then a.. .three or four sentence summary and then I can sort it any which way. 

And so this is still in the data gathering stage so I’m still organizing it different 
ways and rolling it around in my head until some sort of story emerges. 

Analysts occasionally identified, based on local agency or command requirements 
and standard procedures, the requirement to populate automated analytic tools in lieu of, 
or in addition to, personal storage methods. As Analyst 1 identified “Sometimes I find 
myself spending a lot of time feeding these systems, you know trying to populate them so 
I can actually do the analysis.” Or as Analyst 2 further describes 

We use, we have some situational awareness tools that are sort of databases that 
we can populate, that provide us context. So if I am going through serialized 
message traffic I can use that to see, ok is this information new, does it 
corroborate something, is it part of a trend. 

The use of community analytic tools can be driven by several reasons, as Analyst 


2 further identifies 
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Our office chief... .doesn’t want us to rely on hard copy. Because it’s not 
transferrable. So that is the rule in the office, we are not allowed to print things 
unless it’s absolutely necessary, people are held to that... .we have twice as many 
individuals as desks now so we have no other choice but to sort of use these 
collaborative technologies. 

The transformation that takes place at this point in the process is essentially that 
of data into information relevant to the question arrived at in Ti. As described by the 
analyst, the types of information available to them can literally come in any form, in any 
classification. Regardless of the actants involved and the specific part they play, once the 
analyst has collected the information they need, they move forward in the process to the 
third transformation. 

T3: The Transformation of Information into Evaluated Intelligence. At this 
stage of the analytic process the analyst works within their environment to transform the 
information that was identified, sorted, and stored previously into intelligence they will 
use in later steps. Analysts extract relevant portions of the reporting from the larger 
body, the information is evaluated for its reliability, conflicting information is reconciled, 
and a holistic picture of the environment is compiled. At the completion of this stage the 
analyst knows what information they do, and do not have, and the confidence they have 
in that information. Importantly, as described by the participants this information is 
likely to still exist in discrete, heterogeneous elements, it has not yet been synthesized. 

Merely pulling and consolidating the information is not the end state, analysts are 
responsible for evaluating the information, and in an environment where information is 
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incomplete, sometimes contradictory, and the line between what is fact, conjecture, and 
opinion is blurred, it is up to the analyst to weigh the information in order to come to an 
understanding of the situation in question (ODNI, 2015a, pp. 2-3). The participants 
identified that knowing what the source is, and the strengths and weaknesses of that 
source, while seemingly elementary is nevertheless crucial. A mistake identified to be 
common of new analysts is believing everything they see because it is labeled 
“intelligence.” As Analyst 3 stated “I have had to work on being more skeptical, I think 
that’s been very helpful. Because it took me a long time to appreciate the spectrum of 
information that comes out and not believing everything that I read.” Developing that 
skepticism, and not only skepticism but understanding the background of the reporting 
that is forming the basis of any future judgments is key to the transformation taking 
place. Analyst 3 goes on to say “.. .just becoming familiar with the report, and how it 
reads and who wrote the report and what that means has been -1 think that earns your 
weight in gold, really.” 

Understanding the source of reporting gives insight into how the information can 
be skewed. Analyst 7 identified how knowing the details of how the information is 
collected is important in evaluating the information it contains: 

Patrols.. .won’t get the same kind of information that you would get from a 
HUMINT [Human Intelligence] collector say, obviously the patrol leader is 
focused on different things and not collecting intelligence. But at the same time 
he’s going to be able to tell you, ‘hey, this road is passable and this one isn’t’ or 
‘the people in this village tend to be more friendly.’ And sometimes that’s a truer 
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picture than you might see as a collector because the collector, especially if you’re 
engaged in compensating your sources in any way, then you may question what 
you’re told. But as a patrol leader it’s just a guy in a uniform with a rifle and the 
local population’s reaction to that is obviously going to be different... part of it is 
trial and error, just learning which sources are more reliable than others. As I said 
there is always information that you always pretty much take with a grain of salt, 
if it’s coming from a local, and especially if it’s coming from a local official, then 
definitely take it with two grains of salt, always. 

Knowledge of your analytic portfolio is another key element of evaluating the 
information you have, again as related by Analyst 7 

.. .once you get to know your area well enough you can kind of read something 
and say ‘Ok, that makes sense, I can see this person planning that kind of event, I 
can see that kind of relationship being factual.’ And, but at the same time you’ll 
read other things which make no sense whatsoever... .and I guess it’s a little like 
the Sherlock Holmes quote, you know, eliminate the impossible and whatever 
remains is more likely than not to be the truth. 

Analysts also described a trap that can come with regional or functional expertise 
as it relates to evaluating reporting, and the tensions that can arise working with other 
analysts. The introduction of biases, specifically ignoring or filtering information that on 
the surface does not appear to fit an expected norm, was commonly identified. As 
Analyst 1 stated “I’ve run into that argument a lot where you’ll ask people why they think 
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that, you know what is your source for that, what is your... ‘well I just knew that, I’ve 
been doing it for 30 years.’” Or as Analyst 6 relates 

I have this question about [country], well go find the [country] analyst, she’s been 
here forever... .1 don’t want to ask her because I know what answer she’s going to 
give me, it’s no big deal, don’t worry about it. And I want somebody to confirm 
that I think it is a big deal (laughs)... .1 think the more you’re exposed to the more 
you’ll be willing to question those assumptions that need to be questioned. And 
maybe you’re right to do so, and maybe you’re not, but at least it’s been done. 
Reconciling conflicting information into the overall understanding of the situation 
is also part of this evaluation process. As Analyst 4 describes 

.. .all in all you just want to take it in and then say ‘why wouldn’t that be 
plausible’ what tells me it isn’t plausible. And if it’s just the source, you have to 
make different buckets, of ‘I definitely believe this because I read a million things 
that say the same thing’ and then ‘well this seems very plausible but this is new’ 
and then the stuff that says ‘I’m not so sure about it for varying reasons.’ 

Inability to check sources can lead to a bias in trusting classified information, as 
Analyst 6 states: 

.. .in some cases classified information is given a little too much weight because 
you can’t actually check it against anything... .It’s very difficult to be critical of 
something that you almost can’t fact check very easily or very rapidly... .it results 


in a bit of group think. 
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I interpreted this phase of the process alternatively, or perhaps more accurately 
sequentially, described both as a solitary and collaborative process. Initially the analysts 
described that they needed to develop an understanding of reporting in their own head, to 
make sense of the information. At that point the idea that talking over the meaning of 
reporting with other analysts, including those not necessarily sharing the same portfolio, 
was helpful. Analyst 4 describes situations where analysts “would just talk about ‘I read 
these two things’ or ‘I read these three things and I’m trying to figure what it all means. ’” 
While describing the work environment Analyst 1 experienced over the years the 
availability of other analysts or subject matter experts influences how much collaboration 
occurs at this stage 

I think it’s better here for me now that I’m in Washington then when I was in the 
field. I had the same processes there but it was obviously much harder to 
communicate... .here you can just sit down and tell people what you’re thinking, 
it’s easier to meet here. 

At the end of this transformation the analysts describe that they are left with an 
understanding of what they have in the way of evidence, in the form of all-source 
reporting, that is relevant to the question they are attempting to answer. They have 
evaluated the information as to its plausibility, the reliability of the sources providing the 
information, have a clearer understanding of what gaps still remain, and have identified 
any ambiguities or conflicts in the information. The analyst is now ready to move on to 


the next stage of the process. 
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T4: The Transformation of Evaluated Intelligence into Answers. At this stage, 
the analysts describe that the evaluated intelligence is crafted into an answer to the 
question developed in the first transformation. This is when the analyst draws their 
conclusions based on their synthesis of the evidence; and correspondingly identifies gaps 
in knowledge which may preclude them from a more complete answer. Analyst 4 
describes this phase as a mostly solitary activity. 

And.. .then so you read. You know, you have to read your stuff.... I would 
highlight those areas that I really like. And then I would copy and paste them out 
into a separate document where it would be like a running list of notes. Those are 
the things that are either going to directly influence my writing or something that I 
actually want to include, paraphrased or literally verbatim... 

The relevant information collected in T 2 and evaluated in T 3 needs to be available, 
and a space, whether real, digital, or psychic, needs to be found to synthesize it into an 
answer. Again, Analyst 4 continues 

I have a pretty decent amount of space, to spread out all the paperwork and be 
able to organize my research. I’ve got a great amount of storage space where I 
can organize. I have a whiteboard and a space for some maps, maps are very 
important to me. Literally, they tell you where things are (chuckles), and it’s nice 
to have that as a reference. I like the whiteboard, I like being able to just draw up 
some.. .like I’ll put a couple of names here, connect them, keep adding on as my 
research grows, and aside from looking at a computer it’s nice just having this big 
white space that I can just think through. 



Several analysts described the automated tools available to assist analysts at this 
stage. While all were generally appreciative of the support tools could provide, there was 
no consensus on one being better than any other, or on being able to truly automate the 
analytic process. The best tools were those that appeared to assist the analyst in then- 
personal process rather than attempt to be a replacement for it. As Analyst 4 identifies: 
We use Palantir 4 and I like it.. .but I do miss Analyst Notebook 5 . And what I 
miss.. .is being able to draw my own link chart. It’s not about the presentation, 
it’s about the fact that when I create that icon and write the name in it and draw 
the link I remember it. 

The continued introduction of new or competing tools was in fact viewed as a 
distraction in this phase. Analyst 5 described it as 

.. .we are constantly being bombarded with the latest and greatest tools, so it’s like 
the tool of the month club it seems. Suddenly everyone thinks that we should be 
doing everything in TAC 6 or the next month everything should be in NGT 7 or 
whatever it’s called. Or everything is in Palantir, and it’s like you know, it’s 
constant, I mean it’s constant creating new things that are going to solve all our 
problems. 


4 Described by the developer, Palantir Technologies, as an open and extensible software platform 
integrating structured and unstructured data and giving users access to analytic applications. See 
https://www.palantir.com/solutions/intelligence/ 

5 Described by the developer, IBM Analytics, as a visual intelligence environment that allows analysts to 
collate, analyze, and visualize data from disparate sources. See http://www- 
03.ibm.com/software/products/en/analysts-notebook-family 

6 Think Analyze Connect (TAC) is an application that sorts and searches through data streams, provides 
persistent data querying, and supports collaborative analysis and decision making. 

7 Next Generation Trident (NGT) is described by the Chief Technology Officer of the CIA as the agency’s 
analytic tool available to the IC. See https://fcw.com/articles/2010/09/27/feat-gus-hunt-cia-qanda.aspx 
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Analyst 2 described similar frustrations with analytic tools 
Yeah, we’ve tried all sorts of stuff, wikis, and many of the other things that the IC 
has been pushing at one point or another that have kind of withered on the vine. 
We have a basic wiki that’s kind of our home page... 

Another tool related issued identified by several participants and exemplified by 
Analyst 6 is the lack of training associated with the fielding of new tools. 

Software, just like the hardware, is very bureaucratic in the way it occurs. You 
know we just started using One Note for example.. .but that’s existed since at 
least 2007. So here it is 10 years later and we’re discovering it only because our 
commander told us to, yet I know of it from previous places, I think it’s a strong 
tool but there’s no training to tell me how to use it so I really can’t advocate on its 
behalf. 

Concerning tools and this phase of the analytic process, Analyst 1 concisely 
articulated a consensus opinion by stating: 

.. .1 haven’t seen any distinct difference where I’ve taken the effort of using all the 
tools that are available, using these expensive million dollar link diagram tools, 
and then you write a paper; or if I just had a bunch of reports laying on my desk 
with scratch notes... .I’ve written papers both ways and you look at them and it’s 
exactly the same thing. One isn’t any better than the other. 

Regardless of issues with conflicting information, tools, and data of unknown 
reliability, at some point the analyst must provide an answer to the question. A critical 
factor in knowing when it was time to draw the research to a conclusion was time. As 




90 


Analyst 6 succinctly stated . .how do you know when you have enough.. .obviously you 
have enough when the ops guy needs the plan done.” Analyst 7 describes it similarly 

I’ve never really had the luxury of running down, say the origin of stolen cars that 
are being used to support the insurgents. I don’t usually have the physical space 
to be able to do that kind of thing, I don’t usually have the analytic team to divert 
the manpower, and I don’t have the time; because a lot of it is ‘I’ve got a patrol 
going out in five hours I need to know about XYZ village. ’ Or we’ve got the 
nightly update brief to the commander I need you to go in there and put in those 
things that we need for the update. 

The effects of conflicting information or intelligence gaps on developing an 
answer to the question are not lost on the analysts, as Analyst 7 said “.. .there’s been 
plenty of times where I’ve had to say you know what, this isn’t going to be an awesome 
piece, but it’s good enough. And you just take what you have and roll out from there.” 

As the answer to the question is being developed, the analyst is also in a position 
to more precisely identify gaps in knowledge about the issue; rarely can everything be 
known about a subject and questions aren’t asked about established facts. These 
information gaps are fed back to the collection phase of the intelligence cycle, continuing 
the process and, ideally, leading to new information being added to the community’s 
databases. Analyst 4 relates “.. .you have to be able to draw some conclusion from what 
you’ve seen. At that point you identify, as you draw your conclusions, where your gaps 
prevent you from putting a full, comprehensive, cohesive, analytical judgment together.” 
Similarly Analyst 1 states 
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... [I would] try to see if there was a means of answering those questions, you 
know is there enough collection to even address the question. And if not then 
obviously try and draft some sort of requirement to get those gaps filled. 

At the completion of this transformation the analyst is relatively certain that they 
have the best answer to question being asked given the constraints of their environment. 
At this point the answer may still be a conceptual one in the analyst’s mind, but there are 
many physical artifacts (paper, whiteboard, office automation, digital tools, etc.) that 
have captured a portion of the thought process, and which play a role in the next 
transformation. An additional product of this phase is the identification of intelligence 
gaps, which are a by-product of the analytic process and leave the black box as a separate 
output. 

T5: The Transformation of the Answers into an Intelligence Product. In the 

last transformative step in analytic process, the answer arrived at in the previous phase is 
packaged into a finished product, meeting individual agency and IC standards for 
production and tradecraft. If used, graphics are incorporated; if available editors review 
for clarity and grammar; senior analysts and/or managers review for utility and the proper 
agency perspective; and the product may also be formally coordinated with other 
elements of the IC. The participants in the study described a wide variety of intelligence 
products, the spectrum ranging from a face to face brief with minimal documentation to a 
formally published assessment involving editors and graphic support; indicative of the 
wide range of environments analysts work in. This intelligence product, along with the 
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gaps identified in T4, are the outputs of the black box, the completion of the analytic 
process. 

All participants described the actual writing and production of a finished product 
as a process in itself. Although there were commonly identified steps and entities 
participating in the transformation, there was no standardized routine. The range of 
experiences related in the narratives span the breadth of possibilities. Abbreviated, time 
sensitive processes could be found in the range of IC elements from battlefield and 
national agency environments; however longer, more drawn out experiences were only 
found at larger organizations with a wider customer base and larger supporting 
bureaucracy. For example Analyst 5 describes the perspective from a national 
perspective 

First I have to write it, so I have to write the piece. And that tends to be sort of a 
long, drawn out thing. And again different people have different ways. I’m very 
much an outliner, so I’ll just start outlines, and actually using the Microsoft Word 
outline tool and just filling it in, and adding more and more detail, and more and 
more detail. Then we have the world’s worst production routine here, where I’ll 
write it, then it will go to my team leader, and he’ll review it, and I’ll respond to 
comments. Then it will go to the local.. .people here.. .they have comments and I 
respond to that. Then it goes to the [the next higher level], then after that it goes 
to [the local headquarters] and they send it out for IC review and coordination. 


Then after it comes back from that it goes to the actual [headquarters] folks in DC 
who review it, and there’s two different levels of that. So if you say each one of 
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these is at least two or three weeks. So it’s like a six month review process, and 
it’s just horrible. But eventually things get published. 

On the other end of the spectrum, Analyst 6 described the process at the tactical 

level 

.. .when I worked very tactically, you know battalion or brigade level of the 
Army, if I had something I knocked on the XO’s [executive officer] door or even 
the commander’s door and said ‘hey sir, this is important give me five minutes 
and I’ll tell you what’s going on.’.. .the closer you are to the fight the easier that 
access is and the format is very informal... 

Regardless of level the analysts worked at, the writing process was similar for 
many study participants. Analyst 1 echoed Analyst 5 on the use of outlines stating 

I really just open an outline in word and just start typing. I have found for me that 
the best way to get things kick started, is to put my ideas down on paper and start 
writing them down, and start fleshing those things out and changing them as the 
evolve... .It’s a process that builds on itself. 

Analyst 1 goes on to say that outlining is only the beginning of the writing 
process, and describes the desire to write well 

The art of writing something succinct in easy language is actually harder than it 
looks. People might think it’s easier but it’s counterintuitive, it’s actually easier 
to write a long paper than it is to write a short one... .If you only have a minute 
and your key point is buried on page 20 they’re not going to get it. And that’s 
something that I don’t think most analysts are taught very well, is that you can 
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have a really good argument but if your customer doesn’t read it then all the work 
you’ve done was a moot point. 

Analyst 4 had a similar perspective and on the need to communicate effectively 
.. .you need to figure out what the important points are, you need to outline it, put 
it into a construct and then you need to start writing it.. .what determines if it will 
be good or not is how the communication turns out to be... .if you’re not able to 
articulate it then it doesn’t matter. It doesn’t go anywhere. 

While sometimes onerous, the study participants understood why the review 
process was necessary. On one hand corporate review was necessary because the result 
of the analysis, while a personal, or perhaps team, endeavor and achievement, is actually 
a corporate product. As Analyst 5 stated “It took me a while to get my head around that 
I’m not writing [I am assessing] the following. I am writing [the agency] assesses the 
following.” 

The need for outside review and criticism was also seen as important, as Analyst 
3 stated 

.. .when you put out a product for coordination you want people to criticize you as 
much as possible because it protects you and also protects the agency that’s 
publishing it. If you’re making a false assumption or if you put a bias in your 
product that you didn’t perceive you want that to be identified as soon as 
possible.. .you know you can be as trained in critical thinking skills as you want 
but you will never catch every analytical issue on your own. 
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And further . .sometimes the success of a product just involves coordinating it 
with the right person.” Most participants identified that being able to take criticism was a 
necessary component to being a successful analyst. As Analyst 1 stated that during the 
review process 

.. .frankly people are calling into question what you’re doing. So you have to be 
able to defend your work - not yourself, that’s the thing. It’s not an attack on 
you, at least it shouldn’t be. It’s an attack on your hypothesis, on your analysis. 
And I think it’s the understanding that you don’t always have to be right. You’re 
doing the best you can, your best thoughts, your best effort, and there are always 
going to be people that are smarter than you, or have information you don’t have, 
and you have to grow a thick s ki n. 

While Analyst 3 stated that coordination was useful because “hopefully you get to 
minimize the chance of surprise after publication” actually have a response during 
coordination was not a given, noting that “.. .having somebody in the field to coordinate 
with that’s actually willing to read your stuff and provide comments is kind of rare but 
the holy grail at the same time...” 

Graphics were viewed as critical in conveying information, yet were described as 
an ad hoc part of the analytic process. As Analyst 2 says 

So when it comes to, for instance, an annotated map or some sort of geospatial 
product we do that all in house. We used to have access to an analyst with special 
skills from an office across the road, but we don’t use him anymore. There’s a 
new software tool that allows us to do it ourselves with the same fidelity and 
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that’s what we use. And, actually that’s a bit of a shortfall for us. That’s one of 
those tools that really require training but we don’t get the training very often so 
luckily we’ve always had three or four people around that have enough 
experience and time in the job here to produce that sort of stuff; but a lot of times 
a product will go out without those sort of graphics even though they would be 
really useful, unfortunately because one of those individuals with the skills is out. 
This thought echoed by Analyst 6 who said 

I can show you more in a picture than I can in 10 or 15 pages but very few people 
have the tools, or even if they do the skillset to make that picture and translate it 
into something a planner or commander can use. 

A commonly mentioned problem was that it was up to the analysts themselves to 
create the graphics for production, without the proper tool sets or training. As Analyst 3 
stated 

.. .we do have graphics help but it’s more imagery, not necessarily ‘make this 
chart.’.. .there is a graphics shop but it seems more for the.. .front office and events and 
stuff like that. It does not seemed to be geared or readily accessible for intelligence 
production... 

While Analyst 3 also stated the observation that “none of my products have really 
called for software.. .or graphics beyond what one could make in PowerPoint or Excel.” 
Analyst 1 relates the advantage in having access to a graphics capability 

.. .before I came here to headquarters I was doing things myself in PowerPoint 
trying to create graphics that way and they were very, amateur hour. There are 
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some very talented people here in DC who do some really cool things. You give 
them data they can create really cool graphics out of it. That’s a very useful 
function. I know I wouldn’t know how to do that. It’s an art form, it really is, 
taking disparate pieces of information and graphically putting it together for a 
customer. Our customers, I would say almost 100% of the time appreciate the 
graphics more than they do these tomes of data we analysts can put together. 
Despite some literature that suggests dissemination of the analytic product is an 
element of analytic tradecraft (U.S. Army, 1994) analysts are not necessarily integrated 
with the dissemination process and few of the participants identified any role in 
dissemination once the product was complete. Analyst 3 identified that at the combatant 
command level the commander’s read book was the primary and immediate place where 
analytic products went, but after that . .1 actually have zero insight as far as who it goes 
to or how it’s received.” Analyst 5 states that “.. .[the agency] publishes things 
on... JWICS.. .and my general impression is that most people out there don’t know [the 
web page] exists.” Analyst 5 went on to say “I will let me contacts know [I’ve completed 
a product].. .and when I get a publication out I will try and post it on i-Space.” 

The formatting used for the final product, unless at the tactical level where 
analysts personally communicate with their primary customer, is dictated by the higher 
level headquarters, in accordance with community standards. As Analyst 6 describes, “I 
work at a joint command, very high level focus, but our analytical processes, including 
formatting, that sort of thing, are dictated through the intelligence community.” 
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The participants related that many actants were involved in this final 
transformation. It became obvious in the accounts that actants outside the process of 
arriving at the answer, such as external reviewers, management, editors, etc., were 
involved in producing the final product. To navigate this transformation point the analyst 
had to shift from what had generally been a solitary series of phases into a group effort, 
requiring successful interaction with each of the actors. 

Developing the Actor-Network of Intelligence Analysis 

With the participants having described analysis from their perspective, I will now 
trace their understanding of the analytic process and identify the actors taking part in the 
transformations. Figure 6 is a graphic depiction of the analytic process as described by 
the participants, and the first illu mi nation of the black box at the macro level. Adhering 
to the three principles of ANT, that is agnosticism, symmetry, and free association, I will 
continue to develop the actor-network by expanding the transformation points and 
identifying the actors. 


| OUTPUTS 



Figure 6. Actor-network trace within the analytic black box. 
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Ti: The Transformation of Requirements into Questions. In this first 
transformation the analysts described an environment where multiple actants came 
together to change broad requirements into specific questions that the analysts could 
focus on and answer. Many administrative and bureaucratic actants external to the 
analysts own work unit were included in this transformation. In addition to the requestor 
(whether in the form of an institution or an individual) and the analyst, other actors 
identified were: 

• The IC element that the analyst works in, specifically the capabilities and 
limitations of the organization to answer the question. 

• Legislated or administrative policy requirements. 

• DNI guidelines and directives that manage IC activities and establish 
responsibilities for organizations in the IC; see for example ICD 204, National 
Intelligence Priorities Framework (ODNI, 2015b), and ICD 900, Integrated 
Mission Management (ODNI, 2013c). 

• Managers, who play a significant role in the developing production 
requirements and ensuring they are met. 

• The role and requirements of the requestor, as interpreted by analysts and their 
management chain. 

• Other analysts and co-workers which provide opinions and judgments. 

• News and other media streams which influence the perception of ongoing 


situations and impact priorities. 
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• Current events, which may or may not be covered in the news or social media 
but which influence requirements. 

• Time available, by managers and analysts, to address an issue. 

Figure 7 below provides a graphic portrayal of the actants involved in this 

transformation. While not every participant identified each actant, the graphic provides a 
compilation of the narratives and a summary of actants described at a level appropriate 
for this study’s research question. Not shown in the graphic due to the aggregated nature 
of the portrayal is the interaction of the actants between themselves, the tensions as each 
actant seeks to exert force onto the process, and necessarily on the other actors. 



Figure 7. Detailed perspective of actants in Ti. 
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Ti: The Transformation of Reporting into Information. During the second 
transformation the analysts described an environment where tools and technology take a 
more prominent role. Building on the results of the first transformation, analysts can now 
mine the potential of the data available to them by searching and pulling from it relevant 
information. This requires the analyst to interact with a variety of actants including: 

• Information networks and workstations, at different classification levels, each 
requiring their own operating systems, passwords, etc. While analysts 
identified the three primary networks 8 by name, to simplify the graphic (see 
Figure 8) they were consolidated into one icon. 

• Databases, residing on each of the three networks. Again, to simplify the 
graphic the concept of numerous databases at different classification levels 
were consolidated into one icon. 

• Database interfaces and aggregators, several identified as available to IC 
analysts, again consolidated into one icon. 

• Boolean logic, the almost universal language for any database interface, its 
strengths, limitations, analyst proficiency, and the ability to save and store 
queries. 

• Printers, and other standard office tools such as pens, highlighters, paper, etc. 

• Personal data management tools, from the narratives of this study those tools 
the analysts had available to them and chose to use to store the information 

8 Non-secure Internet Protocol Router Network (NIPRNET); Secret Internet Protocol Router Network 
(SIPRNET); and the Joint Worldwide Intelligence Communication System (JWICS). 
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they pulled from the databases, which ran the gamut from Excel spreadsheets 
and Word documents to hardcopy printouts on a desk with handwritten notes. 

• Corporate or community data management or automated analytic tools. 

• Peers, assisting in identifying relevant databases or other sources of 
information. 

• Supervisors and organizational staff, also available to not only identify 
relevant databases or other sources of information, but occasionally in the role 
of gatekeeper to those other sources. 

• Subject matter experts, broadly grouped and categorized in the graphic as 
individuals with specific knowledge on a subject of interest to the analyst, 
which could range from soldiers returning from a patrol to academic or 
industry experts. 

Figure 8 is a graphic portrayal of this segment of the actor-network. It includes 
the caveat that the figure does not include the force of the actants on each other and is a 


composite portrayal of the seven participants. It graphically portrays the point of 
increasing technology and the need for data management on the analytic process. 
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Figure 8. Detailed perspective of actants in T 2 . 


T3: The Transformation of Information into Evaluated Intelligence. In this 
transformation the analysts described the process by which they evaluate the information 
they retrieved in the previous stage. As described by the participants, the analyst is the 
primary actant in this stage, leveraging the additional actants to achieve the best results. 
As described in the narratives, the following actants were identified: 

• Background on the sources of information collected. 

• Specifics on how the information was collected. 

• Peers and other co-workers. 


Particular regional or functional analytic portfolio nuances. 
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• Community biases. 

• Individual biases. 

Not portrayed in Figure 9 is that the analyst is the primary actant in this stage. 
While in T 2 the analyst relied predominately on technological interfaces to realize the 
potential of stored data as relevant information; in T 3 the analysts’ rely on their own 
experience and knowledge of the nuances of the collection methods. As described in the 
narratives, there may not be any tangible output at the completion of this transformation, 
the evaluation of the information may exist solely in the analysts’ own mind. 



Figure 9. Detailed perspective of actants in T3. 
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T 4 : The Transformation of Evaluated Intelligence into Answers. In this 
transformation, the totality of the intelligence information and its assigned credibility is 
synthesized into an answer to the question. Analysts described a process where they 
immerse themselves in the data, in all forms, in order to draw their conclusions. As 
described by the analysts, unless specifically designed as a group project this 
transformation occurs with the analyst and their environment as the primary actants. In 
addition to the analyst these actants include: 

• Information networks and workstations, again at various levels of 
classification depending on the requirements. 

• Data management and office automation tools, such as Word, Excel, and 
PowerPoint. 

• Printers, and other standard office tools such as pens, highlighters, paper, etc. 

• Automated analytic tools, as a category. 

• A physical work environment conducive to reading and notetaking, including 
such items desks, offices, whiteboards, and maps. 

• Available time. 

Figure 10 is a graphic representation of this phase, with the caveats that it does 
not include the relationships between the actants, and that is a composite portrayal of the 
seven participants. I also identify in the graphic the analysts’ comments that there are 
frequently information gaps that remain following the development of an answer, which 
leave the analytic black box and return to feed collection requirements in the intelligence 


cycle. 
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Figure 10. Detailed perspective of actants in T 4 . 


Ts: The Transformation of the Answers into an Intelligence Product. In this 
last transformation, as the answer is crafted into a product, many of the organizational 
actants involved in the first transformation return to influence the creation of the final 
product. Additionally the participants described the introduction of outside analysts as 
reviewers of their draft during the product’s coordination. The list of actants interacting 
with the analyst to create the final product include: 

• DNI guidelines and directives that manage IC production, such as ICD 203 
(ODNI, 2015a), Analytic Standards, ICD 206 (ODNI, 2015c), Sourcing 
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Requirements for Disseminated Analytic Products, and ICD 208 (ODNI, 
2017), Maximizing the Utility of Analytic Products. 

• Managers and senior analysts, to review the product for utility and to ensure it 
represents the correct corporate position. 

• Peers and other analysts, as part of the review process. 

• External agency analysts, as part of the review process. 

• Agency support staff, such as editors, graphic artists, or Geographic 
Information System specialists. 

• Information networks and workstations, again at various levels of 
classification depending on the requirements. 

• Office automation tools, such as Word and PowerPoint. 

Figure 11 graphically portray the actants described by the participants in this 
transformation. As noted with previous graphics, I did not include the relationships 
between the actants since it is a composite portrayal of the seven participants’ narratives. 
Similarly, not every participant identified each actant within their organization. 



108 



Conclusions 

While Figure 6 provides the macro level tracing of the actor-network, Figures 7 
through 11 provide actant level detail of the analytic process. Derived from the 
narratives of the study participants, it portrays the complex mixture of actants that 
participate in the analytic process; those entities that play a role in the transformation of 
requirements and reporting into intelligence products and refined information gaps. It is 
this understanding of the analytic process that I used to address the research questions 
driving this study. 
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The first question, how does an analyst move along the process of analysis, from 
start to finish, is captured graphically in Figure 6. The graphic portrays the five specific 
points along the process where the analyst, in conjunction with other actants, transforms 
the requirements and reporting into the finished intelligence product. 

The graphic captures an idealized process, distilled from the experiences of seven 
analysts and not representative of any particular example. Situations may arise that cause 
loops in the process, for example if newly collected reporting is made available that 
causes the analyst to return from T 4 to T 2 . In other time sensitive cases, or when the 
analyst has developed an expertise in their portfolio, the transformations may be 
abbreviated to the point that they may seem to occur almost simultaneously. Regardless 
of how compressed or expanded however, these five transformations constitute the 
analytic process. 

The first sub-question, how the analytic process is learned, was described as a 
combination of formal training, on-the-job trial and error experiences, and intuition. All 
analyst participants described being exposed to some sort of formal analytic training, and 
they related that they found that training useful. The participants also stated that there 
was no comprehensive training program that prepared an analyst to know, let alone be 
successful at, everything they were expected to know and do. The participants identified 
gaps in training; and accepted that some knowledge, such as which databases were 
available and what they contained, were things that couldn’t be taught in a classroom 
environment. Many participants also described that being a “good analyst” required 
more than training, that certain personality types, specifically those comfortable in 
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dealing with uncertainty and those naturally inquisitive, were more successful at 
intelligence analysis than others. 

The second sub-question, concerning variances between participants in their 
descriptions of the analytic process, appear to be rooted in the specific environment or 
situation the analyst is working in rather than the process itself. For example, while 
coding the individual interview transcripts the concept of management influence in the 
process was universal, but the specifics of the layers of management or the degree of 
influence was specific to the agency or organization. As I discussed previously, there 
may also be loops in the transformations, or the steps may be abbreviated, however the 
process as described remained consistent among participants. Given the deliberate 
variety in work experiences in the sample selected, I expected variances in the actants, 
such as sources of information available, level of coordination needed, or types of 
products created. Even with these work unit specific variances however, the role of those 
actants in the trace of the process remained consistent. 

The second question concerned what software, hardware, tools, people, etc., 
populate the analysts environment, and how does the analyst negotiate that environment. 
Figures 7-11 present a visualization of the actants involved in the analytic process from 
the analyst’s perspective, aggregated to a common IC level. What is made apparent by 
using the ANT approach to the question, and what is crucial to understand, is that an 
analyst is not the only actant in the analytic process. Further, the analyst may not be the 
primary actor at any one transformation point. Precisely how an analyst negotiates their 
relationship with other actants in the process is personal and case specific, as I captured 
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and portrayed in the narrative extracts provided. From the perspective of improving 
tradecraft, and analysis more generally, understanding these relationships between actants 
during the transformations is key. Knowing the actants involved and understanding that 
there are tensions between them as the transformations take place also provide the answer 
to the first sub-question, it is clear from the narratives that the environment does affect 
the analytic process. 

The second sub-question, can the analyst manipulate the environment to match 
their needs or will the environment determine the process, is again case specific, and 
depends on the relative strength of the other actants in the process. Realizing that there 
are other actors involved, and that they have a role in the negotiation during the 
transformations that take place, is a key finding of this study and the meaningful answer 
to this question. Even if the analyst is the most forceful actant in a particular 
transformation, there are still compromises made by all participants in the negotiation. 

The third question, what skills do analysts need to successfully navigate their 
environment, goes to the heart of improving analysis and analytic tradecraft. Previous 
models of analysis, specifically Kent’s 1949 model, focused on the analyst as the sole 
actor in the process. Analytic tradecraft standards (DNI, 2007a; ODNI, 2015a) sustained 
that model without expanding it, and focused on the final stage of the process, production 
and presentation. The skills and personality traits the analyst needs to be successful are 
those required to negotiate the transformations and move on to the next step in the 
process. As described in the narratives these can include but are not limited to 
proficiency in the office automation tools provided, time management, interpersonal 
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communications and conflict management, information on collection systems, knowledge 
on issues relevant to their customers, and writing and graphic presentation. Further, these 
skills and knowledge should not be treated as separate and distinct entities, but as parts of 
an entire tradecraft package tied together in one social environment. 

Evaluation of Trustworthiness 

Credibility 

As I described in Chapter 3, the credibility of this study is related to procedures 
taken to ensure the best possible population, methodology, and data collection techniques 
were used to address the research questions. The research design I developed for this 
study incorporated efforts to enhance the credibility of the findings. In this section I 
address each element of credibility. 

The population I recruited and selected for this study was limited to all-source 
intelligence analysts currently involved in the production of intelligence products within 
the U.S. IC. Individuals that volunteered as participants who had prior experience as 
analysts, but were now in positions of support staff or management, were not selected. 
While valuable insights might still be gained from individuals involved in support and 
leadership roles, or with prior experience as analysts, by focusing on those currently 
creating intelligence products I selected a population most likely to provide the most 
current and relevant narratives. 

The methodology I selected, a narrative study using ANT as the conceptual 
framework, was used to drive to the heart of the research questions. Rather than deciding 
a priori what constitutes analytic tradecraft and evaluating how well analysts accomplish 
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those tasks, participants described in their own voice the process of analysis as they 
accomplish it. Then, by using ANT as the guide to identify and trace the transformation 
of requirements and information into finished intelligence products, I developed a trace 
of the process previously veiled by the black box of analysis. 

The data collection techniques I used, recorded interviews conducted in person, 
by phone, or skype, provided the most credible method of gaining a true narrative of the 
analytic process. My use of the interview guide in a semi-structured environment 
allowed the participants the freedom to talk through the analytic process, to tell their own 
stories about how they conduct their craft. Member checks allowed each participant the 
opportunity to review their interview and follow up with additional insights or thoughts. 
Transferability 

Transferability is analogous to generalizability or external validation in 
quantitative studies. As I described in Chapter 3, purposeful case sampling was used to 
obtain a wide range of experiences across the U.S. IC, a bureaucratic construct that 
stretches from national level intelligence agencies informing long term policy decisions 
to tactical and operational level military units and local and state level fusion centers 
(ODNI, 2013b). I purposely selected participants to enhance the transferability of the 
findings. 

A mix of DoD and non-DoD national level intelligence agencies were represented 
by the participants (see Appendix B). I achieved a similar ratio of DoD and non-DoD 
participants as is the ratio of agencies in the IC. Similarly, participant experiences and 
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current assignments span the gamut from producing intelligence for tactical military units 
deployed in Iraq and Afghanistan through intelligence support for national policy makers. 

The level of analysis I selected for this study, a meso-understanding of the 
analytic process, lends itself to transferability across the IC. The broad categories of 
inputs (requirements and intelligence data) and outputs (intelligence products and refined 
intelligence gaps) are captured in the intelligence cycle (see Figures 1 & 2) and standard 
across the IC, enhancing applicability with individual components. In order to maximize 
transferability within the IC, a complete ANT guided mapping of specific actants and 
interactions between them was forfeited. It is my experience that individual agencies 
have wide latitude on applying community directives and establishing their own policies 
and procedures; therefore a more detailed agency or site specific assessment of the 
actants involved would have limited transferability. 

Dependability and Confirmability 

I enhanced dependability, or the ability to replicate findings, in this study by the 
use of member checks, an interview guide, and the MAXQDA 12 qualitative data 
analysis tool. My use of member checks ensured that I accurately transcribed the 
recording of the interview and captured analytic process as lived by the participants. The 
interview guide provides a means for other researchers to replicate the interviews with 
other participants. My use of MAXQDA 12 created an electronic trace of the 
development of the coding scheme and identification of the translation points. My use of 
recorded and member checked interviews and MAXQDA 12 also provide a strong degree 
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of confirmability, that is the ability of other researchers to confirm or corroborate my 
findings, as described by Trochim (2006). 

Summary 

Figures 6-11 provide a graphic representation of the intelligence analysis 
process previously black boxed, identifying in detail for the first time since Kent (1949) 
how requirements and reporting are transformed into finished intelligence products and 
knowledge gaps. These findings provide an understanding of that process from the 
perspective of current analysts, identifying the other actants that analysts must interact 
with during the analytic process. Latour and Woolgar (1979) provided an innovative 
understanding of how “facts” are constructed in laboratories by demystifying the 
relationships between the actants in that environment. In this study, I provided the basis 
for a more comprehensive understanding of intelligence analysis as currently practiced in 
the U.S. IC by allowing analysts to describe in their own words what they do and using 
ANT as the framework to explore and map that process. 

Through their narratives the participants established that intelligence analysis is a 
social enterprise resulting in knowledge creation, and is neither a purely mechanical 
process nor a solitary intellectual one. While Law (1992) found test tubes and scanning 
electron microscopes in the Salk Institute, the narratives in this study reveal an 
environment that includes notebooks and pencils, whiteboards and spreadsheets, and 
information networks spanning several layers of classification. The analyst is only one 
member of this society which is comprised of a heterogeneous mix of people, databases, 
and policies as illustrated in Figures 7-11. And as is true of all societies, each member 
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has a role in defining and shaping each other (Law, 2007). Disruptions to the status quo, 
a failure of any one member of the group in their assigned responsibility, introduction of 
new members, or a shift in relative power between actants, will have a disruptive effect 
on the entire enterprise. 

As the stated goal of IC reform and the establishment of analytic tradecraft 
standards is better analysis (ODNI, 2015a), leading, therefore to better intelligence 
products, this actor-network perspective of analysis can provide the starting point to 
expand the community’s understanding of what is meant by tradecraft. Further, it allows 
for targeted reforms, and for unintended or unexpected corollary effects of changes in the 
process to be identified and considered. With this enhanced understanding of the analytic 
process, I will provide specific recommendations for implementing these insights and 
implications in the Chapter 5. 
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Chapter 5: Discussion, Conclusions, and Recommendations 

Introduction 

My purpose in conducting this research was to develop a better understanding of 
the process of intelligence analysis as a necessary component of improving analytic 
tradecraft. I designed the study taking a narrative approach using actor-network theory as 
the conceptual framework (Latour, 2005). Using semistructured interviews with analysts 
currently working in the IC I asked how they created intelligence products in their work 
environment. By developing the actor-network of the analytic process, tracing the 
transformations of the inputs and identifying the actants involved in those 
transformations, a clearer understanding of the skills needed to successfully navigate that 
process was developed. 

I identified five discrete transformation points in the narratives, where actants 
working together resulted in a change of state to the inputs of the analytic process (see 
Figure 6). I named these points: Ti, Requirements => Questions; T2, Reporting => 
Information; T3, Information => Evaluated Intelligence; T4, Evaluated Intelligence => 
Answers; T5, Answers => Products. Next, I derived from the narratives the complex 
mixture of actants that participate in the analytic process in addition to the analyst (see 
Figures 7-11). It is this combination of actants operating in the social environment of 
the analyst that transform requirements and reporting into intelligence products and 
refined information gaps. 

Through their narratives the participants established that intelligence analysis is 
neither a purely mechanical process nor a solitary intellectual one. The narratives in this 
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study reveal a complex environment of people, policies, equipment, and information. 

This detailed understanding of the analytic process can provide the starting point to 
expand the intelligence community’s understanding of what is meant by tradecraft, 
allowing for identification of actual deficiencies and targeted reforms. 

Interpretation of the Findings 

In Chapter 2 I established that the lack of a clear and comprehensive 
understanding of the analytic process is acknowledged by scholars and practitioners as an 
impediment to improving analysis. Further, I demonstrated that earlier attempts to 
identify and fix analysis, and standardize tradecraft and corresponding tradecraft 
standards, were a result of an a priori determination of flaws in the process at a particular 
moment of history. Despite the effort put into implementing community-wide analytic 
tradecraft standards following passage of the IRTPA (ODNI, 2015 a), I found there to be 
no noticeable improvement in the quality of analysis as measured by the rate of generally 
accepted intelligence failures. 

The trace of the analytic process (see Figure 6) and the identification of the 
actants involved in that process (see Figures 7-11) provide a rigorous conception of 
what occurs during intelligence analysis in the U.S. IC. Consistent with other ANT 
guided research efforts (Oh, et. al., 2016; Psenaka, 2008), this study identified from the 
perspective of the analysts the complex social environment in which knowledge, in the 
form of finished intelligence products, is created within the IC. For the first time since 
Kent (1949) adapted his academic approach to research to describe intelligence analysis, 
there is an understanding of the analytic process rooted in research. 
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The research design I chose for this study allowed me to provide a perspective of 
the analytic process that did not filter the narrative experiences of the participants through 
a predetermined framework. The process that occurs during the five transformations is 
intelligence analysis in the U.S. IC. How the analyst negotiates their relationship with 
other actants in the process and the roles they assume during the transformations is 
analytic tradecraft. 

As described in Chapter 2, previous efforts to improve analysis have alternated 
between tweaking the inputs and outputs to the process. From the Hoover Commission’s 
(1955) call for increasing collection through the most recent tradecraft standards (ODNI, 
2015a) focusing on communication to the reader, the most expeditious route to improving 
analysis has always appeared to be through the tangible inputs and outputs of the process. 
Any improvements to analysis using this indirect approach are serendipitous, and also run 
the risk of causing more harm than good (Betts, 1978, pp. 84 - 89). 

With the understanding gained by exploring and mapping the analytic process and 
identifying the actants involved, it is now possible for practitioners and policy makers to 
take a more deliberate approach to addressing analytic tradecraft. Interpreting the 
findings through the lens of improving tradecraft, with the ultimate goal of improving 
analysis, can provide a new approach to future initiatives. The following paragraphs are 
intended as initial suggestions to leverage the benefits an actor-network understanding of 
the analytic process provides. 

Intelligence analysis, and corresponding analytic tradecraft, should be viewed as a 
multi-step process involving several actants, rather than solely as an analyst’s 
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idiosyncratic thought process. The narrative accounts of the participants describe a 
process building on itself where errors early in the process can result in flawed analysis, 
regardless of how well the final product is written or presented. Efforts made to improve 
the skills an analyst needs to navigate through the transformation points should 
incorporate the concept of an analysis as a multistep process, for example explaining the 
effect that query writing has on future transformations, or that file management has on 
proper evaluation of sources. 

Structured analytic techniques and critical thinking (Hall & Citrenbaum, 2010; 
Heuer, 1999; Moore, 2011) are necessary but insufficient tradecraft skills, resident in 
only some transformations and not within all actants. The analysts’ narratives identified 
s ki lls which do not seem directly tied to intelligence analysis, such as time management, 
proficiency in basic office automation, an ability to provide meaningful responses to 
requests for draft product reviews, and interpersonal communication as equally 
necessary. Addressing these skills in the context of the analytic process can likely 
provide the most benefit. 

The participants’ narratives identified how the relationship between the requestor 
and the analyst was the foundation of the analytic process. As the first transformation in 
the process, how the requestor’s requirements are articulated, understood, and 
transformed into analytic questions is crucial, the more aligned the analytic questions are 
with the requirements the more likely the results of the analytic process are to be useful to 
the requestor. A clearer and shared understanding of the mission of the requester among 
all the actants in Ti, that is knowing why the requestor wants the information, how it will 
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be used, and what effects it will have on decision making, will establish a path to more 
useful intelligence products. 

At the agency and production center level the integration of graphic production 
into the analytic process and a clearer definition of the analyst’s role in the production of 
graphics is needed. The IC as a whole recognizes the importance of quality graphics, 
evidenced by the addition of effective visual information as an evaluated tradecraft 
standard (ODNI, 2015a, pp. 4-5) and through the narratives of the analysts themselves. 
However the narratives also reveal the ad hoc nature of graphics in the analytic process, 
frustration with obtaining graphic support, and the lack of tools and training for analysts 
to do it themselves. 

Limitations of the Study 

The actor-network described in Figure 6 and expanded in Figures 7 - 11 is a 
composite of narratives drawn from a wide variety of analytic assignments and 
experiences in accordance with the research design. The U.S. IC is a large community, 
with “thousands” of currently serving analysts spread out over 17 agencies (ODNI, 
2006b, p. 12). While I believe this trace of the analytic process is a sound community 
overview, my work with the individual narratives indicates to me that agency specific 
studies are needed to provide a detailed identification of specific actants (software used, 
availability of support staff, role of management, etc.) and their relationships during the 
transformations. 

During the recruitment process, in the consent forms, and prior to the actual 
interviews, all participants were reminded of the unclassified nature of this study. 
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However, the social environment that an all-source analyst works in, is by definition of 
the sources used, a classified one. Given that the narratives used as the basis for the 
actor-network were provided in this unclassified environment, there is no way me to 
determine what, if any, information concerning the analytic process or the specific actants 
involved in that process, was left out by the participants. 

Recommendations for Future Research 
My use of ANT to arrive at a generalized description of a community of social 
networks, instead of the more commonly used extensive study of an individual case 
(Latour, 2005, pp. 173 - 1740, proved successful in addressing my research questions. 
While aggregating the data from the different participants during coding precluded a 
detailed analysis of individual actors and the relationships between them, it provided a 
necessary foundation for understanding the analytic process. My adoption of this 
approach provides several opportunities for further research on analysis and analytic 
tradecraft in the intelligence community. Specific examples include: 

• Agency specific studies using ANT as a guiding framework, conducted in 
a classified environment and with participant non-attribution, to further 
refine the actants in the analytic process and to gain fidelity on the 
relationships between actants during specific transformations. 

• Using the actor-network trace of the analytic process as a guide (see 
Figures 6 - 11), a community-wide or agency specific review of available 
training, professional development, or structured mentorship could 
identify gaps in what analysts are expected to do and opportunities they 
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have to learn those skills. Specific attention can be paid to those cross¬ 
transformation skills analysts need to navigate the process itself, not just 
the discrete actions within it. 

• Conduct post-hoc studies at the agency or community wide level at the 
necessary classification for both intelligence failures and successes. Using 
the actor-network trace of the analytic process as a guide, patterns of 
activity within the transformations may become visible which may lead to 
focused improvements in training or processes. 

• Conduct a similar study on single source analysts, e.g., geospatial 
intelligence (GEOINT) or signals intelligence (SIGINT) analysts, to 
determine unique characteristics in those fields and corresponding areas 
for targeted improvements. 

Implications 

Intelligence Analysis and Tradecraft 

The goal of all intelligence community reforms is to improve the analysis 
provided to the customer, and the latest attempts to codify and improve tradecraft has 
been no exception (Collins, 2004). However, success has been elusive. As I identified in 
Chapter 2, previous attempts to define and contextualize tradecraft, that is how analysts 
navigate the analytic process, have been limited. The last known attempt to describe 
intelligence analysis, Kent’s 1949 model, was analyst centric, based on an academic 
environment, and created long before the current information and technology heavy 
environment a current analyst operates in could even be conceived. Current ODNI 
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tradecraft standards were initially developed in response to allegations of politicization of 
intelligence during the Cold War, and are written to provide a means to evaluate the 
presentation of the analytic output (MacEachin, 1994). They put the majority of the 
analysts’ efforts to meet those standards and receive satisfactory tradecraft evaluations at 
the final step in the analytic process. 

By using the words of the analysts themselves, this study provides an IC specific 
description of the analytic process, providing the initial illumination of that black box. It 
also provides a roster of the actants, those entities that exert influence on the process, 
playing a role in the shaping of the product. Understanding how an analyst navigates that 
environment provides a more complete understanding of analytic tradecraft and has 
implications for the profession at the national, agency, and individual levels. 

At the national level, where IC wide policies and procedures are developed, 
approaching intelligence analysis conceptually as a multistage process with multiple 
actants would change the approach to analytic tradecraft. With the additional definition 
on the analytic process that this study provides, the current focus on the end product, the 
output of the process, could be expanded. As a community of practitioners, tradecraft 
standards could be developed that include the actions that take place within the 
transformations as reporting is turned into intelligence products. These expanded 
tradecraft standards could also address the other key actants in the process, and 
acknowledge analysis as the social endeavor the participants described it as. 

At the individual agency and production center level, where managers and staff 
have a direct role in defining the environment of the analyst and the other actants, the 
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understanding of the analytic process this study provides allows for better informed 
decision making. This actor-network trace of analysis provides them a roadmap to 
identify the tangible effects their decisions have. These leaders now have a framework 
that allows them to conceptualize the ripples that occur throughout the analytic process 
when decisions changing policies, organizational structures, or analytic tools are made. It 
also provides a tool to identify the other actants involved in the analytic process that may 
either act to enhance or diminish the intended results of decisions these leaders make. 

At the individual analyst level, the mapping of the analytic process this study 
developed can serve as a self-assessment tool. Current tradecraft standards were 
designed to grade the final product of analysis (MacEachin, 1994, p.13). The actor- 
network trace of analysis provides analysts a more comprehensive tool to appraise their 
individual skills, how they navigate the transformation points, and where they may need 
to become more involved with other actants in the process. 

Actor-Network Theory and Tradecraft 

As I described in Chapter 2, the use of ANT in the study of knowledge creation 
and in organizational studies is well established. The utility of ANT in the study of 
intelligence analysis was unproven, however previous studies in laboratories, classrooms, 
and other government agencies provided me confidence in its applicability. ANT has, 
however, primarily been used in individual case studies. By focusing on one event, one 
agency, ANT provides a means to not only identify the individual actants, but to explore 
the relationship between them, the translations of power that occur as each exerts its 
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influence and enlists allies during the transformations (Latour, 2005). My use of ANT to 
explore a community of semiautonomous agencies generally accomplishing the same 
function and generally ascribing to the same operating standards and guidelines was 
untested. 

It was my assessment that the combination of the lack of any rigorous study of the 
analytic process for almost 60 years, the scale of the U.S. IC, and the implementation of 
community wide tradecraft standards resulted in the need for an IC level understanding of 
the analytic process as a first step to my developing a better understanding of analytic 
tradecraft. This approach precluded my use of an individual agency as a case study 
which might have more limited generalizability. My use of ANT as the conceptual 
framework in a study across several agencies and different narrative experiences was an 
innovative use of the approach. The similarity in describing the analytic process by the 
diverse participants does provide me confidence in the description and graphic 
representation of the analytic process and accompanying actants. However, the ability to 
dive deeper into the individual actants and their relationships was sacrificed to provide 
this meso-level IC perspective. This study provides an example of how ANT can be used 
beyond individual case studies and the benefits it can provide, but also the limitations that 
result from it. 

Conclusion 

As a result of this research effort, I have demonstrated that analytic tradecraft as it 
is currently understood in the IC and captured in DNI directives is relevant to only a 
small portion of the analytic process. Additionally, current tradecraft standards do not 
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address other actants that have a role in analysis, at some points in the process possibly a 
greater role than the analyst. By tracing the actor-network of the analytic process, I have 
provided in this study a starting point for more tailored and comprehensive efforts to 
improve analytic tradecraft, and intelligence analysis more generally. The implications 
for potential use of these results exist at the IC management, agency, and individual 
analyst level. 
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Appendix A: Interview Questions and Guide 
The questions in the interview guide will generally follow Kent’s (1949) analytic 
process with the intent of ensuring information concerning the complex ecology of 
analysis is being elicited from analysts throughout the process. A sampling of possible 
questions follows. 

• Tell me how you develop the idea or theme for an intelligence product, 
what drives your decisions and what factors do you have to keep in mind? 

• Once you have an idea for a product where do you start looking for 
information on it? What forms does the information come in, for example 
digital, hardcopy, orally...? 

• How do you know when you have enough information, is it a conscious 
decision to stop collecting new information and start creating the analytic 
product? Do you communicate with anyone about information they may 
have, or the extent of information you’ve collected. Have you been 
contacted before by someone else creating an analytic product looking for 
information? 

• How do you collect, compile, evaluate, and synthesize the information you 
gather - what kind of notes do you keep? Do you use paper, white boards, 
software or other digital tools? Who do you communicate with during this 
process, if anyone? 

• How do you get your analysis out to your consumer, briefings, written 
products, posts, phone calls, etc.? Do you know ahead of time how you 
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will present the analysis, how much of the presentation format guides your 
work? 

Tell me about your work environment, your work space? What equipment 
is available to you on your desktop, what do you use, what additional kind 
of tools or equipment do you have access to or use? 

Do you ever take you work outside your work environment, in a separate 
work area, a conference room, cafeteria, home, etc.? Why? 
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Appendix B: Participant Matrix 


Study ID 

Current IC 

agency 

Years 

analytic 

experience 

Personnel 

status 

Interview 

means 

Interview 

date 

Analyst 1 

Non-DoD IC agency 

20 

Civilian 

In person 

August 2015 

Analyst 2 

Combatant 

Command 

18 

Civilian 

Skype 

September 2015 

Analyst 3 

Combatant 

Command 

6 

Civilian 

Phone 

August 2015 

Analyst 4 

Non-DoD IC agency 

17 

Civilian 

In person 

August 2015 

Analyst 5 

Non-DoD IC agency 

5 

Civilian 

Phone 

June 2015 

Analyst 6 

Combatant 

Command 

12 

Active military 

Phone 

March 2016 

Analyst 7 

DoD Element 

5 

Active military 

Phone 

March 2016 
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Appendix C: Coding and Analysis Matrix 


Code 

Theme 

Ti 

t 2 

t 3 

t 4 

t 5 

Data gathering 

Stages 

3 

6 

7 

5 

4 

Production 

Stages 

6 

6 

1 

4 

13 

Requirements 

Stages 

14 

2 

0 

3 

3 

Dissemination 

Stages 

0 

1 

0 

0 

6 

Classification 

Process 

0 

1 

1 

1 

0 

Team brainstorm 

Process 

1 

4 

4 

3 

2 

Analytic process 

Process 

11 

24 

9 

22 

22 

Collection 

Process 

0 

2 

1 

0 

1 

Available assets 

Process 

0 

0 

2 

2 

0 

External assets 

Process 

0 

2 

1 

1 

2 

Team capabilities 

Process 

0 

3 

0 

2 

3 

Capabilities 

Tools 

2 

0 

0 

1 

2 

Tools 

Tools 

0 

14 

6 

13 

10 

Work environment 

Tools 

1 

10 

4 

6 

9 

Agency differences 

Role 

0 

1 

0 

1 

2 

Analyst’s role 

Role 

18 

21 

22 

19 

50 



